T12b_Hadoop - Simple YARN job for Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc67fe0-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68184-495a-11e8-a14e-0242ac130002",
     "previous": null
    }
   },
   "source": [
    "# About: Simple YARN job for Test\n",
    "\n",
    "---\n",
    "\n",
    "YARN, HDFSの動作確認として、Piを実行してみる。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc68184-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc682ec-495a-11e8-a14e-0242ac130002",
     "previous": "9bc67fe0-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "## *Operation Note*\n",
    "\n",
    "*This is a cell for your own recording.  ここに経緯を記述*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc682ec-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc6844a-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68184-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "# Notebookと環境のBinding\n",
    "\n",
    "Inventory中のgroup名でBind対象を指示する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc6844a-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc685a8-495a-11e8-a14e-0242ac130002",
     "previous": "9bc682ec-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "target_group = 'hadoop_all_testcluster'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc685a8-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc686f2-495a-11e8-a14e-0242ac130002",
     "previous": "9bc6844a-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "# MapReduceサンプルジョブの実行\n",
    "\n",
    "クライアントマシン(`hadoop_client`グループのもの)で、 `yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 10 1000` のように実行してみる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc686f2-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68850-495a-11e8-a14e-0242ac130002",
     "previous": "9bc685a8-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/tmpgwiJOL'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tempfile\n",
    "work_dir = tempfile.mkdtemp()\n",
    "work_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9bc68850-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc689a4-495a-11e8-a14e-0242ac130002",
     "previous": "9bc686f2-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "yarn_result = os.path.join(work_dir, 'yarn-result')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc689a4-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68aee-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68850-495a-11e8-a14e-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Number of Maps  = 10\r\n",
      "Samples per Map = 1000\r\n",
      "Wrote input for Map #0\r\n",
      "Wrote input for Map #1\r\n",
      "Wrote input for Map #2\r\n",
      "Wrote input for Map #3\r\n",
      "Wrote input for Map #4\r\n",
      "Wrote input for Map #5\r\n",
      "Wrote input for Map #6\r\n",
      "Wrote input for Map #7\r\n",
      "Wrote input for Map #8\r\n",
      "Wrote input for Map #9\r\n",
      "Starting Job\r\n",
      "Job Finished in 29.128 seconds\r\n",
      "Estimated value of Pi is 3.1408000000000000000016/08/19 10:36:56 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/19 10:36:56 INFO input.FileInputFormat: Total input paths to process : 10\r\n",
      "16/08/19 10:36:57 INFO mapreduce.JobSubmitter: number of splits:10\r\n",
      "16/08/19 10:36:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1471559811619_0002\r\n",
      "16/08/19 10:36:58 INFO client.YARNRunner: Number of stages: 2\r\n",
      "16/08/19 10:36:59 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=XXX.XXX.XXX.2.4.2.0-258, revision=fa554fdce4e3495e09a310e0a32bb34ccc5946ad, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160425-0638 ]\r\n",
      "16/08/19 10:36:59 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/19 10:36:59 INFO client.TezClient: Submitting DAG application with id: application_1471559811619_0002\r\n",
      "16/08/19 10:36:59 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs://hdfs-cluster//apps/tez/tez.tar.gz\r\n",
      "16/08/19 10:36:59 INFO client.TezClient: Tez system stage directory hdfs://hdfs-cluster/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/.tez/application_1471559811619_0002 doesn't exist and is created\r\n",
      "16/08/19 10:37:00 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1471559811619_0002, dagName=QuasiMonteCarlo\r\n",
      "16/08/19 10:37:00 INFO impl.YarnClientImpl: Submitted application application_1471559811619_0002\r\n",
      "16/08/19 10:37:00 INFO client.TezClient: The url to track the Tez AM: http://testvm001:8088/proxy/application_1471559811619_0002/\r\n",
      "16/08/19 10:37:00 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/19 10:37:01 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/19 10:37:01 INFO mapreduce.Job: The url to track the job: http://testvm001:8088/proxy/application_1471559811619_0002/\r\n",
      "16/08/19 10:37:01 INFO mapreduce.Job: Running job: job_1471559811619_0002\r\n",
      "16/08/19 10:37:06 INFO mapreduce.Job: Job job_1471559811619_0002 running in uber mode : false\r\n",
      "16/08/19 10:37:06 INFO mapreduce.Job:  map 0% reduce 0%\r\n",
      "16/08/19 10:37:13 INFO mapreduce.Job:  map 10% reduce 0%\r\n",
      "16/08/19 10:37:15 INFO mapreduce.Job:  map 80% reduce 0%\r\n",
      "16/08/19 10:37:19 INFO mapreduce.Job:  map 90% reduce 0%\r\n",
      "16/08/19 10:37:23 INFO mapreduce.Job:  map 100% reduce 100%\r\n",
      "16/08/19 10:37:23 INFO mapreduce.Job: Job job_1471559811619_0002 completed successfully\r\n",
      "16/08/19 10:37:23 INFO mapreduce.Job: Counters: 0\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a \"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 10 1000\" -l { target_group } hadoop_client \\\n",
    "         | tee {yarn_result}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc68aee-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68c42-495a-11e8-a14e-0242ac130002",
     "previous": "9bc689a4-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "# 実行経過の確認"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc68c42-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68d96-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68aee-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "## ログの確認\n",
    "\n",
    "ログは、 `yarn logs` コマンドにより確認することができる。この際、Application IDが必要。この情報はログから確認することができる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc68d96-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc68eea-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68c42-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'application_1471559811619_0002'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yarn_application_id = None\n",
    "with open(yarn_result, 'r') as f:\n",
    "    for line in f:\n",
    "        if 'Submitting DAG application with id' in line:\n",
    "            yarn_application_id = line.split()[-1]\n",
    "assert(yarn_application_id is not None)\n",
    "yarn_application_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc68eea-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc6903e-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68d96-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "ログを確認してみる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc6903e-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69188-495a-11e8-a14e-0242ac130002",
     "previous": "9bc68eea-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "\n",
      "\n",
      "Container: container_e01_1471559811619_0002_01_000001 on testvm004_45454\n",
      "==========================================================================\n",
      "LogType:dag_1471559811619_0002_1.dot\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:1002\n",
      "Log Contents:\n",
      "digraph QuasiMonteCarlo {\n",
      "graph [ label=\"QuasiMonteCarlo\", fontsize=24, fontname=Helvetica];\n",
      "node [fontsize=12, fontname=Helvetica];\n",
      "edge [fontsize=9, fontcolor=blue, fontname=Arial];\n",
      "\"QuasiMonteCarlo.initialmap_MRInput\" [ label = \"initialmap[MRInput]\", shape = \"box\" ];\n",
      "\"QuasiMonteCarlo.initialmap_MRInput\" -> \"QuasiMonteCarlo.initialmap\" [ label = \"Input [inputClass=MRInputLegacy,\\n initializer=]\" ];\n",
      "\"QuasiMonteCarlo.finalreduce_MROutput\" [ label = \"finalreduce[MROutput]\", shape = \"box\" ];\n",
      "\"QuasiMonteCarlo.initialmap\" [ label = \"initialmap[MapProcessor]\" ];\n",
      "\"QuasiMonteCarlo.initialmap\" -> \"QuasiMonteCarlo.finalreduce\" [ label = \"[input=OrderedPartitionedKVOutput,\\n output=OrderedGroupedInputLegacy,\\n dataMovement=SCATTER_GATHER,\\n schedulingType=SEQUENTIAL]\" ];\n",
      "\"QuasiMonteCarlo.finalreduce\" [ label = \"finalreduce[ReduceProcessor]\" ];\n",
      "\"QuasiMonteCarlo.finalreduce\" -> \"QuasiMonteCarlo.finalreduce_MROutput\" [ label = \"Output [outputClass=MROutputLegacy,\\n committer=MROutputCommitter]\" ];\n",
      "}\n",
      "End of LogType:dag_1471559811619_0002_1.dot\n",
      "\n",
      "LogType:directory.info\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:14982\n",
      "Log Contents:\n",
      "ls -l:\n",
      "total 56\n",
      "-rw-r--r-- 1 yarn yarn  210 Aug 19 10:37 container_tokens\n",
      "-rwx------ 1 yarn yarn  782 Aug 19 10:37 default_container_executor.sh\n",
      "-rwx------ 1 yarn yarn  728 Aug 19 10:37 default_container_executor_session.sh\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.jar -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\n",
      "lrwxrwxrwx 1 yarn yarn  117 Aug 19 10:37 job.split -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\n",
      "lrwxrwxrwx 1 yarn yarn  125 Aug 19 10:37 job.splitmetainfo -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/15/job.splitmetainfo\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.xml -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/16/job.xml\n",
      "-rwx------ 1 yarn yarn 5564 Aug 19 10:37 launch_container.sh\n",
      "lrwxrwxrwx 1 yarn yarn  119 Aug 19 10:37 tez-conf.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/tez-conf.pb\n",
      "lrwxrwxrwx 1 yarn yarn  118 Aug 19 10:37 tez-dag.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-dag.pb\n",
      "lrwxrwxrwx 1 yarn yarn  138 Aug 19 10:37 tez.session.local-resources.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/tez.session.local-resources.pb\n",
      "lrwxrwxrwx 1 yarn yarn   60 Aug 19 10:37 tezlib -> /hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\n",
      "drwx--x--- 2 yarn yarn 4096 Aug 19 10:37 tmp\n",
      "find -L . -maxdepth 5 -ls:\n",
      "655667    4 drwx--x---   3 yarn     yarn         4096 Aug 19 10:37 .\n",
      "655670    4 -rw-r--r--   1 yarn     yarn           12 Aug 19 10:37 ./.container_tokens.crc\n",
      "655672    4 -rw-r--r--   1 yarn     yarn           52 Aug 19 10:37 ./.launch_container.sh.crc\n",
      "655660  108 -r-x------   1 yarn     yarn       108179 Aug 19 10:37 ./job.xml\n",
      "655669    4 -rw-r--r--   1 yarn     yarn          210 Aug 19 10:37 ./container_tokens\n",
      "655675    4 -rwx------   1 yarn     yarn          782 Aug 19 10:37 ./default_container_executor.sh\n",
      "655450    4 drwxr-xr-x   3 yarn     yarn         4096 Aug 18 23:45 ./tezlib\n",
      "655479  272 -r-xr-xr-x   1 yarn     yarn       275362 Apr 25 15:39 ./tezlib/tez-mapreduce-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655482  152 -r-xr-xr-x   1 yarn     yarn       154464 Apr 25 15:39 ./tezlib/tez-tests-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655478  532 -r-xr-xr-x   1 yarn     yarn       542434 Apr 25 15:39 ./tezlib/tez-runtime-library-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655480   48 -r-xr-xr-x   1 yarn     yarn        47672 Apr 25 15:39 ./tezlib/tez-examples-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655477  164 -r-xr-xr-x   1 yarn     yarn       164267 Apr 25 15:39 ./tezlib/tez-runtime-internals-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655487    8 -r-xr-xr-x   1 yarn     yarn         6334 Apr 25 15:41 ./tezlib/tez-yarn-timeline-cache-plugin-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655485   64 -r-xr-xr-x   1 yarn     yarn        62943 Apr 25 15:41 ./tezlib/tez-history-parser-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655476   68 -r-xr-xr-x   1 yarn     yarn        67195 Apr 25 15:38 ./tezlib/tez-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655488   24 -r-xr-xr-x   1 yarn     yarn        22125 Apr 25 15:42 ./tezlib/tez-yarn-timeline-history-with-fs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655484   28 -r-xr-xr-x   1 yarn     yarn        25622 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655489    4 drwxr-xr-x   2 yarn     yarn         4096 Aug 18 23:45 ./tezlib/lib\n",
      "655528  524 -r-xr-xr-x   1 yarn     yarn       533455 Apr 23 05:17 ./tezlib/lib/protobuf-java-2.5.0.jar\n",
      "655535   20 -r-xr-xr-x   1 yarn     yarn        16560 Apr 23 08:05 ./tezlib/lib/api-asn1-api-1.0.0-M20.jar\n",
      "655542 1444 -r-xr-xr-x   1 yarn     yarn      1475955 Apr 23 06:44 ./tezlib/lib/htrace-core-3.1.0-incubating.jar\n",
      "655525  976 -r-xr-xr-x   1 yarn     yarn       995968 Apr 23 05:17 ./tezlib/lib/snappy-java-XXX.XXX.XXX.1.jar\n",
      "655548  128 -r-xr-xr-x   1 yarn     yarn       130458 Apr 23 05:45 ./tezlib/lib/jersey-client-1.9.jar\n",
      "655533  676 -r-xr-xr-x   1 yarn     yarn       691479 Apr 23 08:05 ./tezlib/lib/apacheds-kerberos-codec-2.0.0-M15.jar\n",
      "655552    8 -r-xr-xr-x   1 yarn     yarn         4467 Apr 23 05:19 ./tezlib/lib/aopalliance-1.0.jar\n",
      "655516   64 -r-xr-xr-x   1 yarn     yarn        62050 Apr 23 05:34 ./tezlib/lib/commons-logging-1.1.3.jar\n",
      "655569  152 -r-xr-xr-x   1 yarn     yarn       153798 Apr 25 14:41 ./tezlib/lib/hadoop-azure-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655497   80 -r-xr-xr-x   1 yarn     yarn        81743 Apr 23 09:18 ./tezlib/lib/jettison-1.3.4.jar\n",
      "655557  312 -r-xr-xr-x   1 yarn     yarn       319099 Apr 23 05:19 ./tezlib/lib/okhttp-2.4.0.jar\n",
      "655565  848 -r-xr-xr-x   1 yarn     yarn       865838 Apr 23 05:19 ./tezlib/lib/jackson-databind-2.2.3.jar\n",
      "655566   36 -r-xr-xr-x   1 yarn     yarn        33483 Apr 23 05:19 ./tezlib/lib/jackson-annotations-2.2.3.jar\n",
      "655545  104 -r-xr-xr-x   1 yarn     yarn       105112 Apr 23 05:17 ./tezlib/lib/servlet-api-2.5.jar\n",
      "655520  144 -r-xr-xr-x   1 yarn     yarn       143602 Apr 23 05:17 ./tezlib/lib/commons-digester-1.8.jar\n",
      "655574   36 -r-xr-xr-x   1 yarn     yarn        34835 Apr 25 14:37 ./tezlib/lib/hadoop-yarn-server-web-proxy-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655531  424 -r-xr-xr-x   1 yarn     yarn       433368 Apr 23 07:01 ./tezlib/lib/httpclient-4.2.5.jar\n",
      "655576  100 -r-xr-xr-x   1 yarn     yarn        98771 Apr 25 14:58 ./tezlib/lib/hadoop-yarn-registry-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655567 11672 -r-xr-xr-x   1 yarn     yarn     11948376 Apr 23 08:19 ./tezlib/lib/aws-java-sdk-1.7.4.jar\n",
      "655572 1500 -r-xr-xr-x   1 yarn     yarn      1533812 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-core-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655490   12 -r-xr-xr-x   1 yarn     yarn         8869 Apr 23 06:23 ./tezlib/lib/slf4j-log4j12-1.7.5.jar\n",
      "655517  572 -r-xr-xr-x   1 yarn     yarn       582965 Apr 23 05:19 ./tezlib/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar\n",
      "655573  736 -r-xr-xr-x   1 yarn     yarn       751873 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655580  700 -r-xr-xr-x   1 yarn     yarn       714194 Apr 23 05:37 ./tezlib/lib/javassist-3.18.1-GA.jar\n",
      "655529  188 -r-xr-xr-x   1 yarn     yarn       190432 Apr 23 08:06 ./tezlib/lib/gson-2.2.4.jar\n",
      "655550  696 -r-xr-xr-x   1 yarn     yarn       710492 Apr 23 05:19 ./tezlib/lib/guice-3.0.jar\n",
      "655507  148 -r-xr-xr-x   1 yarn     yarn       147952 Apr 23 05:44 ./tezlib/lib/jersey-json-1.9.jar\n",
      "655570  652 -r-xr-xr-x   1 yarn     yarn       666763 Apr 23 05:19 ./tezlib/lib/azure-storage-2.2.0.jar\n",
      "655554  184 -r-xr-xr-x   1 yarn     yarn       185401 Apr 25 15:01 ./tezlib/lib/hadoop-yarn-client-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655563  532 -r-xr-xr-x   1 yarn     yarn       540751 Apr 23 05:19 ./tezlib/lib/jetty-6.1.26.hwx.jar\n",
      "655503   16 -r-xr-xr-x   1 yarn     yarn        15010 Apr 23 05:17 ./tezlib/lib/xmlenc-0.52.jar\n",
      "655579  336 -r-xr-xr-x   1 yarn     yarn       340884 Apr 23 05:19 ./tezlib/lib/fst-2.24.jar\n",
      "655544 1672 -r-xr-xr-x   1 yarn     yarn      1710975 Apr 25 14:57 ./tezlib/lib/hadoop-yarn-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655506  576 -r-xr-xr-x   1 yarn     yarn       588337 Apr 23 05:37 ./tezlib/lib/commons-collections-3.2.2.jar\n",
      "655527   96 -r-xr-xr-x   1 yarn     yarn        94672 Apr 23 05:17 ./tezlib/lib/xz-1.0.jar\n",
      "655541  268 -r-xr-xr-x   1 yarn     yarn       270342 Apr 23 06:44 ./tezlib/lib/curator-recipes-2.7.1.jar\n",
      "655555  736 -r-xr-xr-x   1 yarn     yarn       751238 Apr 23 09:18 ./tezlib/lib/commons-collections4-4.1.jar\n",
      "655502 1564 -r-xr-xr-x   1 yarn     yarn      1599627 Apr 23 06:43 ./tezlib/lib/commons-math3-3.1.1.jar\n",
      "655495  184 -r-xr-xr-x   1 yarn     yarn       185140 Apr 23 05:19 ./tezlib/lib/commons-io-2.4.jar\n",
      "655512  228 -r-xr-xr-x   1 yarn     yarn       232248 Apr 23 05:28 ./tezlib/lib/jackson-core-asl-1.9.13.jar\n",
      "655530   72 -r-xr-xr-x   1 yarn     yarn        71570 Apr 25 14:27 ./tezlib/lib/hadoop-auth-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655499 3420 -r-xr-xr-x   1 yarn     yarn      3498444 Apr 25 14:46 ./tezlib/lib/hadoop-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655491   28 -r-xr-xr-x   1 yarn     yarn        26084 Apr 23 05:34 ./tezlib/lib/slf4j-api-1.7.5.jar\n",
      "655534   44 -r-xr-xr-x   1 yarn     yarn        44925 Apr 23 08:05 ./tezlib/lib/apacheds-i18n-2.0.0-M15.jar\n",
      "655543 2060 -r-xr-xr-x   1 yarn     yarn      2108699 Apr 25 14:55 ./tezlib/lib/hadoop-yarn-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655521  188 -r-xr-xr-x   1 yarn     yarn       188671 Apr 23 05:14 ./tezlib/lib/commons-beanutils-1.7.0.jar\n",
      "655501   44 -r-xr-xr-x   1 yarn     yarn        41123 Apr 23 05:14 ./tezlib/lib/commons-cli-1.2.jar\n",
      "655510   24 -r-xr-xr-x   1 yarn     yarn        23346 Apr 23 05:14 ./tezlib/lib/stax-api-1.0-2.jar\n",
      "655556 8228 -r-xr-xr-x   1 yarn     yarn      8421957 Apr 25 14:50 ./tezlib/lib/hadoop-hdfs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655538 1172 -r-xr-xr-x   1 yarn     yarn      1199572 Apr 23 05:47 ./tezlib/lib/netty-3.6.2.Final.jar\n",
      "655509  104 -r-xr-xr-x   1 yarn     yarn       105134 Apr 23 05:17 ./tezlib/lib/jaxb-api-2.2.2.jar\n",
      "655539  184 -r-xr-xr-x   1 yarn     yarn       186273 Apr 23 06:43 ./tezlib/lib/curator-framework-2.7.1.jar\n",
      "655518  192 -r-xr-xr-x   1 yarn     yarn       192699 Apr 23 05:19 ./tezlib/lib/jackson-core-2.2.3.jar\n",
      "655532  224 -r-xr-xr-x   1 yarn     yarn       227275 Apr 23 07:14 ./tezlib/lib/httpcore-4.2.4.jar\n",
      "655562 1024 -r-xr-xr-x   1 yarn     yarn      1045744 Apr 23 05:19 ./tezlib/lib/leveldbjni-all-1.8.jar\n",
      "655494   36 -r-xr-xr-x   1 yarn     yarn        33031 Apr 23 09:18 ./tezlib/lib/jsr305-2.0.3.jar\n",
      "655519  292 -r-xr-xr-x   1 yarn     yarn       298829 Apr 23 05:17 ./tezlib/lib/commons-configuration-1.6.jar\n",
      "655564  104 -r-xr-xr-x   1 yarn     yarn       104578 Apr 25 14:41 ./tezlib/lib/hadoop-aws-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655568  616 -r-xr-xr-x   1 yarn     yarn       627814 Apr 23 08:19 ./tezlib/lib/joda-time-2.9.3.jar\n",
      "655505  268 -r-xr-xr-x   1 yarn     yarn       273370 Apr 23 05:17 ./tezlib/lib/commons-net-3.1.jar\n",
      "655526  236 -r-xr-xr-x   1 yarn     yarn       241367 Apr 23 06:43 ./tezlib/lib/commons-compress-1.4.1.jar\n",
      "655500   20 -r-xr-xr-x   1 yarn     yarn        17371 Apr 25 14:27 ./tezlib/lib/hadoop-annotations-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655540   68 -r-xr-xr-x   1 yarn     yarn        69500 Apr 23 06:43 ./tezlib/lib/curator-client-2.7.1.jar\n",
      "655551    4 -r-xr-xr-x   1 yarn     yarn         2497 Apr 23 05:19 ./tezlib/lib/javax.inject-1.jar\n",
      "655547  448 -r-xr-xr-x   1 yarn     yarn       458739 Apr 23 05:45 ./tezlib/lib/jersey-core-1.9.jar\n",
      "655493 1612 -r-xr-xr-x   1 yarn     yarn      1648200 Apr 23 05:18 ./tezlib/lib/guava-11.0.2.jar\n",
      "655498   28 -r-xr-xr-x   1 yarn     yarn        26514 Apr 23 05:19 ./tezlib/lib/stax-api-1.0.1.jar\n",
      "655571  404 -r-xr-xr-x   1 yarn     yarn       412739 Apr 23 06:23 ./tezlib/lib/commons-lang3-3.3.2.jar\n",
      "655537  776 -r-xr-xr-x   1 yarn     yarn       794058 Apr 25 14:27 ./tezlib/lib/zookeeper-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655523  300 -r-xr-xr-x   1 yarn     yarn       303139 Apr 25 14:28 ./tezlib/lib/avro-1.7.4.jar\n",
      "655549   64 -r-xr-xr-x   1 yarn     yarn        65012 Apr 23 05:19 ./tezlib/lib/guice-servlet-3.0.jar\n",
      "655558   64 -r-xr-xr-x   1 yarn     yarn        64661 Apr 23 05:19 ./tezlib/lib/okio-1.4.0.jar\n",
      "655561  192 -r-xr-xr-x   1 yarn     yarn       194354 Apr 23 05:14 ./tezlib/lib/xml-apis-1.3.04.jar\n",
      "655559 1740 -r-xr-xr-x   1 yarn     yarn      1779991 Apr 23 05:19 ./tezlib/lib/netty-all-4.0.23.Final.jar\n",
      "655504   60 -r-xr-xr-x   1 yarn     yarn        58160 Apr 23 08:05 ./tezlib/lib/commons-codec-1.4.jar\n",
      "655582   60 -r-xr-xr-x   1 yarn     yarn        60819 Apr 25 14:38 ./tezlib/lib/hadoop-yarn-server-timeline-plugins-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655581   44 -r-xr-xr-x   1 yarn     yarn        41755 Apr 23 05:51 ./tezlib/lib/objenesis-2.1.jar\n",
      "655578  244 -r-xr-xr-x   1 yarn     yarn       246782 Apr 25 15:00 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655514   20 -r-xr-xr-x   1 yarn     yarn        18336 Apr 23 08:06 ./tezlib/lib/jackson-jaxrs-1.9.13.jar\n",
      "655513  764 -r-xr-xr-x   1 yarn     yarn       780664 Apr 23 05:17 ./tezlib/lib/jackson-mapper-asl-1.9.13.jar\n",
      "655575  408 -r-xr-xr-x   1 yarn     yarn       413811 Apr 25 14:59 ./tezlib/lib/hadoop-yarn-server-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655546  176 -r-xr-xr-x   1 yarn     yarn       177487 Apr 23 05:19 ./tezlib/lib/jetty-util-6.1.26.hwx.jar\n",
      "655522  204 -r-xr-xr-x   1 yarn     yarn       206035 Apr 23 05:17 ./tezlib/lib/commons-beanutils-core-1.8.0.jar\n",
      "655553   16 -r-xr-xr-x   1 yarn     yarn        14786 Apr 23 05:45 ./tezlib/lib/jersey-guice-1.9.jar\n",
      "655508  872 -r-xr-xr-x   1 yarn     yarn       890168 Apr 23 05:17 ./tezlib/lib/jaxb-impl-2.2.3-1.jar\n",
      "655515   28 -r-xr-xr-x   1 yarn     yarn        27084 Apr 23 05:17 ./tezlib/lib/jackson-xc-1.9.13.jar\n",
      "655577  132 -r-xr-xr-x   1 yarn     yarn       134133 Apr 23 08:04 ./tezlib/lib/servlet-api-2.5-20081211.jar\n",
      "655496  280 -r-xr-xr-x   1 yarn     yarn       284220 Apr 23 05:14 ./tezlib/lib/commons-lang-2.6.jar\n",
      "655492  480 -r-xr-xr-x   1 yarn     yarn       489884 Apr 23 05:17 ./tezlib/lib/log4j-1.2.17.jar\n",
      "655524   32 -r-xr-xr-x   1 yarn     yarn        29555 Apr 23 05:17 ./tezlib/lib/paranamer-2.3.jar\n",
      "655560 1204 -r-xr-xr-x   1 yarn     yarn      1229125 Apr 23 05:14 ./tezlib/lib/xercesImpl-2.9.1.jar\n",
      "655511   64 -r-xr-xr-x   1 yarn     yarn        62983 Apr 23 05:17 ./tezlib/lib/activation-1.1.jar\n",
      "655536   80 -r-xr-xr-x   1 yarn     yarn        79912 Apr 23 08:05 ./tezlib/lib/api-util-1.0.0-M20.jar\n",
      "655486    8 -r-xr-xr-x   1 yarn     yarn         7784 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-with-acls-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655475  868 -r-xr-xr-x   1 yarn     yarn       888761 Apr 25 15:38 ./tezlib/tez-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655481 1196 -r-xr-xr-x   1 yarn     yarn      1222496 Apr 25 15:39 ./tezlib/tez-dag-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655483 1600 -r-xr-xr-x   1 yarn     yarn      1636310 Apr 25 15:40 ./tezlib/tez-ui-XXX.XXX.XXX.2.4.2.0-258.war\n",
      "655674    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor_session.sh.crc\n",
      "655645    4 -r-x------   1 yarn     yarn         1467 Aug 19 10:37 ./job.split\n",
      "655671    8 -rwx------   1 yarn     yarn         5564 Aug 19 10:37 ./launch_container.sh\n",
      "655648  108 -r-x------   1 yarn     yarn       107877 Aug 19 10:37 ./tez-dag.pb\n",
      "655673    4 -rwx------   1 yarn     yarn          728 Aug 19 10:37 ./default_container_executor_session.sh\n",
      "655657    4 -r-x------   1 yarn     yarn          357 Aug 19 10:37 ./job.splitmetainfo\n",
      "655651    4 -r-x------   1 yarn     yarn          753 Aug 19 10:37 ./tez.session.local-resources.pb\n",
      "655654   44 -r-x------   1 yarn     yarn        44350 Aug 19 10:37 ./tez-conf.pb\n",
      "655676    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor.sh.crc\n",
      "655668    4 drwx--x---   2 yarn     yarn         4096 Aug 19 10:37 ./tmp\n",
      "655642  268 -r-x------   1 yarn     yarn       273458 Aug 19 10:37 ./job.jar\n",
      "broken symlinks(find -L . -maxdepth 5 -type l -ls):\n",
      "End of LogType:directory.info\n",
      "\n",
      "LogType:history.txt.appattempt_1471559811619_0002_000001\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:104506\n",
      "Log Contents:\n",
      "{\"entity\":\"tez_application_1471559811619_0002\",\"entitytype\":\"TEZ_APPLICATION\",\"otherinfo\":{\"user\":\"ansible\",\"config\":{\"dfs.block.access.key.update.interval\":\"600\",\"dfs.block.access.token.enable\":\"false\",\"dfs.block.access.token.lifetime\":\"600\",\"dfs.block.scanner.volume.bytes.per.second\":\"1048576\",\"dfs.blockreport.initialDelay\":\"0\",\"dfs.blockreport.intervalMsec\":\"21600000\",\"dfs.blockreport.split.threshold\":\"1000000\",\"dfs.blocksize\":\"134217728\",\"dfs.bytes-per-checksum\":\"512\",\"dfs.cachereport.intervalMsec\":\"10000\",\"dfs.client-write-packet-size\":\"65536\",\"dfs.client.block.write.replace-datanode-on-failure.best-effort\":\"false\",\"dfs.client.block.write.replace-datanode-on-failure.enable\":\"true\",\"dfs.client.block.write.replace-datanode-on-failure.policy\":\"DEFAULT\",\"dfs.client.block.write.retries\":\"3\",\"dfs.client.cached.conn.retry\":\"3\",\"dfs.client.context\":\"default\",\"dfs.client.datanode-restart.timeout\":\"30\",\"dfs.client.domain.socket.data.traffic\":\"false\",\"dfs.client.failover.connection.retries\":\"0\",\"dfs.client.failover.connection.retries.on.timeouts\":\"0\",\"dfs.client.failover.max.attempts\":\"15\",\"dfs.client.failover.proxy.provider.hdfs-cluster\":\"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider\",\"dfs.client.failover.sleep.base.millis\":\"500\",\"dfs.client.failover.sleep.max.millis\":\"15000\",\"dfs.client.file-block-storage-locations.num-threads\":\"10\",\"dfs.client.file-block-storage-locations.timeout.millis\":\"1000\",\"dfs.client.https.keystore.resource\":\"ssl-client.xml\",\"dfs.client.https.need-auth\":\"false\",\"dfs.client.mmap.cache.size\":\"256\",\"dfs.client.mmap.cache.timeout.ms\":\"3600000\",\"dfs.client.mmap.enabled\":\"true\",\"dfs.client.mmap.retry.timeout.ms\":\"300000\",\"dfs.client.read.shortcircuit\":\"false\",\"dfs.client.read.shortcircuit.skip.checksum\":\"false\",\"dfs.client.read.shortcircuit.streams.cache.expiry.ms\":\"300000\",\"dfs.client.read.shortcircuit.streams.cache.size\":\"256\",\"dfs.client.short.circuit.replica.stale.threshold.ms\":\"1800000\",\"dfs.client.slow.io.warning.threshold.ms\":\"30000\",\"dfs.client.use.datanode.hostname\":\"false\",\"dfs.client.use.legacy.blockreader.local\":\"false\",\"dfs.client.write.exclude.nodes.cache.expiry.interval.millis\":\"600000\",\"dfs.datanode.address\":\"XXX.XXX.XXX.0:50010\",\"dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction\":\"0.75f\",\"dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold\":\"10737418240\",\"dfs.datanode.balance.bandwidthPerSec\":\"1048576\",\"dfs.datanode.block-pinning.enabled\":\"false\",\"dfs.datanode.block.id.layout.upgrade.threads\":\"12\",\"dfs.datanode.bp-ready.timeout\":\"20\",\"dfs.datanode.cache.revocation.polling.ms\":\"500\",\"dfs.datanode.cache.revocation.timeout.ms\":\"900000\",\"dfs.datanode.data.dir\":\"\\/hadoop\\/data01\\/dfs\\/datadir\",\"dfs.datanode.data.dir.perm\":\"700\",\"dfs.datanode.directoryscan.interval\":\"21600\",\"dfs.datanode.directoryscan.threads\":\"1\",\"dfs.datanode.dns.interface\":\"default\",\"dfs.datanode.dns.nameserver\":\"default\",\"dfs.datanode.drop.cache.behind.reads\":\"false\",\"dfs.datanode.drop.cache.behind.writes\":\"false\",\"dfs.datanode.du.reserved\":\"0\",\"dfs.datanode.failed.volumes.tolerated\":\"0\",\"dfs.datanode.fsdatasetcache.max.threads.per.volume\":\"4\",\"dfs.datanode.handler.count\":\"10\",\"dfs.datanode.hdfs-blocks-metadata.enabled\":\"false\",\"dfs.datanode.http.address\":\"XXX.XXX.XXX.0:50075\",\"dfs.datanode.https.address\":\"XXX.XXX.XXX.0:50475\",\"dfs.datanode.ipc.address\":\"XXX.XXX.XXX.0:50020\",\"dfs.datanode.max.locked.memory\":\"0\",\"dfs.datanode.max.transfer.threads\":\"4098\",\"dfs.datanode.readahead.bytes\":\"4193404\",\"dfs.datanode.scan.period.hours\":\"504\",\"dfs.datanode.shared.file.descriptor.paths\":\"\\/dev\\/shm,\\/tmp\",\"dfs.datanode.slow.io.warning.threshold.ms\":\"300\",\"dfs.datanode.sync.behind.writes\":\"false\",\"dfs.datanode.transfer.socket.recv.buffer.size\":\"131072\",\"dfs.datanode.transfer.socket.send.buffer.size\":\"131072\",\"dfs.datanode.use.datanode.hostname\":\"false\",\"dfs.default.chunk.view.size\":\"32768\",\"dfs.encrypt.data.transfer\":\"false\",\"dfs.encrypt.data.transfer.cipher.key.bitlength\":\"128\",\"dfs.ha.automatic-failover.enabled\":\"true\",\"dfs.ha.fencing.methods\":\"sshfence\",\"dfs.ha.fencing.ssh.connect-timeout\":\"30000\",\"dfs.ha.fencing.ssh.private-key-files\":\"\\/var\\/lib\\/hadoop-hdfs\\/.ssh\\/id_rsa\",\"dfs.ha.log-roll.period\":\"120\",\"dfs.ha.namenodes.hdfs-cluster\":\"testvm001,testvm002\",\"dfs.ha.tail-edits.period\":\"60\",\"dfs.ha.zkfc.port\":\"8019\",\"dfs.heartbeat.interval\":\"3\",\"dfs.hosts\":\"\\/etc\\/hadoop\\/conf\\/hosts.list\",\"dfs.hosts.exclude\":\"\\/etc\\/hadoop\\/conf\\/hosts.exclude\",\"dfs.http.policy\":\"HTTP_ONLY\",\"dfs.https.server.keystore.resource\":\"ssl-server.xml\",\"dfs.image.compress\":\"false\",\"dfs.image.compression.codec\":\"org.apache.hadoop.io.compress.DefaultCodec\",\"dfs.image.transfer.bandwidthPerSec\":\"0\",\"dfs.image.transfer.chunksize\":\"65536\",\"dfs.image.transfer.timeout\":\"60000\",\"dfs.journalnode.edits.dir\":\"\\/hadoop\\/data\\/jn\",\"dfs.journalnode.http-address\":\"XXX.XXX.XXX.0:8480\",\"dfs.journalnode.https-address\":\"XXX.XXX.XXX.0:8481\",\"dfs.journalnode.rpc-address\":\"XXX.XXX.XXX.0:8485\",\"dfs.namenode.accesstime.precision\":\"3600000\",\"dfs.namenode.acls.enabled\":\"false\",\"dfs.namenode.audit.loggers\":\"default\",\"dfs.namenode.avoid.read.stale.datanode\":\"false\",\"dfs.namenode.avoid.write.stale.datanode\":\"false\",\"dfs.namenode.backup.address\":\"XXX.XXX.XXX.0:50100\",\"dfs.namenode.backup.http-address\":\"XXX.XXX.XXX.0:50105\",\"dfs.namenode.blocks.per.postponedblocks.rescan\":\"10000\",\"dfs.namenode.checkpoint.check.period\":\"60\",\"dfs.namenode.checkpoint.dir\":\"file:\\/\\/\\/hadoop\\/tmp\\/hadoop-ansible\\/dfs\\/namesecondary\",\"dfs.namenode.checkpoint.edits.dir\":\"file:\\/\\/\\/hadoop\\/tmp\\/hadoop-ansible\\/dfs\\/namesecondary\",\"dfs.namenode.checkpoint.max-retries\":\"3\",\"dfs.namenode.checkpoint.period\":\"1800\",\"dfs.namenode.checkpoint.txns\":\"1000000\",\"dfs.namenode.checkpoints.txns\":\"400000\",\"dfs.namenode.datanode.registration.ip-hostname-check\":\"true\",\"dfs.namenode.decommission.blocks.per.interval\":\"500000\",\"dfs.namenode.decommission.interval\":\"30\",\"dfs.namenode.decommission.max.concurrent.tracked.nodes\":\"100\",\"dfs.namenode.delegation.key.update-interval\":\"86400000\",\"dfs.namenode.delegation.token.max-lifetime\":\"604800000\",\"dfs.namenode.delegation.token.renew-interval\":\"86400000\",\"dfs.namenode.edit.log.autoroll.check.interval.ms\":\"300000\",\"dfs.namenode.edit.log.autoroll.multiplier.threshold\":\"2.0\",\"dfs.namenode.edits.dir\":\"\\/hadoop\\/data\\/dfs\\/namedir\",\"dfs.namenode.edits.journal-plugin.qjournal\":\"org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager\",\"dfs.namenode.edits.noeditlogchannelflush\":\"false\",\"dfs.namenode.enable.retrycache\":\"true\",\"dfs.namenode.fs-limits.max-blocks-per-file\":\"1048576\",\"dfs.namenode.fs-limits.max-component-length\":\"255\",\"dfs.namenode.fs-limits.max-directory-items\":\"1048576\",\"dfs.namenode.fs-limits.max-xattr-size\":\"16384\",\"dfs.namenode.fs-limits.max-xattrs-per-inode\":\"32\",\"dfs.namenode.fs-limits.min-block-size\":\"1048576\",\"dfs.namenode.full.block.report.lease.length.ms\":\"300000\",\"dfs.namenode.handler.count\":\"32\",\"dfs.namenode.heartbeat.recheck-interval\":\"300000\",\"dfs.namenode.http-address\":\"XXX.XXX.XXX.0:50070\",\"dfs.namenode.http-address.hdfs-cluster.testvm001\":\"testvm001:50070\",\"dfs.namenode.http-address.hdfs-cluster.testvm002\":\"testvm002:50070\",\"dfs.namenode.https-address\":\"XXX.XXX.XXX.0:50470\",\"dfs.namenode.inotify.max.events.per.rpc\":\"1000\",\"dfs.namenode.invalidate.work.pct.per.iteration\":\"0.32f\",\"dfs.namenode.kerberos.internal.spnego.principal\":\"${dfs.web.authentication.kerberos.principal}\",\"dfs.namenode.kerberos.principal.pattern\":\"*\",\"dfs.namenode.lazypersist.file.scrub.interval.sec\":\"300\",\"dfs.namenode.lifeline.handler.ratio\":\"0.10\",\"dfs.namenode.list.cache.directives.num.responses\":\"100\",\"dfs.namenode.list.cache.pools.num.responses\":\"100\",\"dfs.namenode.list.encryption.zones.num.responses\":\"100\",\"dfs.namenode.max.extra.edits.segments.retained\":\"10000\",\"dfs.namenode.max.full.block.report.leases\":\"6\",\"dfs.namenode.max.objects\":\"0\",\"dfs.namenode.metrics.logger.period.seconds\":\"600\",\"dfs.namenode.name.dir\":\"\\/hadoop\\/data\\/dfs\\/namedir\",\"dfs.namenode.name.dir.restore\":\"false\",\"dfs.namenode.num.checkpoints.retained\":\"2\",\"dfs.namenode.num.extra.edits.retained\":\"1000000\",\"dfs.namenode.path.based.cache.block.map.allocation.percent\":\"0.25\",\"dfs.namenode.path.based.cache.refresh.interval.ms\":\"30000\",\"dfs.namenode.path.based.cache.retry.interval.ms\":\"30000\",\"dfs.namenode.reject-unresolved-dn-topology-mapping\":\"false\",\"dfs.namenode.replication.considerLoad\":\"true\",\"dfs.namenode.replication.interval\":\"3\",\"dfs.namenode.replication.min\":\"1\",\"dfs.namenode.replication.work.multiplier.per.iteration\":\"2\",\"dfs.namenode.resource.check.interval\":\"5000\",\"dfs.namenode.resource.checked.volumes.minimum\":\"1\",\"dfs.namenode.resource.du.reserved\":\"104857600\",\"dfs.namenode.retrycache.expirytime.millis\":\"600000\",\"dfs.namenode.retrycache.heap.percent\":\"0.03f\",\"dfs.namenode.rpc-address.hdfs-cluster.testvm001\":\"testvm001:8020\",\"dfs.namenode.rpc-address.hdfs-cluster.testvm002\":\"testvm002:8020\",\"dfs.namenode.safemode.extension\":\"10000\",\"dfs.namenode.safemode.min.datanodes\":\"0\",\"dfs.namenode.safemode.threshold-pct\":\"0.999f\",\"dfs.namenode.secondary.http-address\":\"XXX.XXX.XXX.0:50090\",\"dfs.namenode.secondary.https-address\":\"XXX.XXX.XXX.0:50091\",\"dfs.namenode.shared.edits.dir\":\"qjournal:\\/\\/testvm001:8485;testvm002:8485;testvm003:8485\\/hdfs-cluster\",\"dfs.namenode.stale.datanode.interval\":\"30000\",\"dfs.namenode.startup.delay.block.deletion.sec\":\"0\",\"dfs.namenode.support.allow.format\":\"true\",\"dfs.namenode.top.enabled\":\"true\",\"dfs.namenode.top.num.users\":\"10\",\"dfs.namenode.top.window.num.buckets\":\"10\",\"dfs.namenode.top.windows.minutes\":\"1,5,25\",\"dfs.namenode.write.stale.datanode.ratio\":\"0.5f\",\"dfs.namenode.xattrs.enabled\":\"true\",\"dfs.nameservices\":\"hdfs-cluster\",\"dfs.permissions.enabled\":\"true\",\"dfs.permissions.superusergroup\":\"supergroup\",\"dfs.replication\":\"3\",\"dfs.replication.max\":\"512\",\"dfs.secondary.namenode.kerberos.internal.spnego.principal\":\"${dfs.web.authentication.kerberos.principal}\",\"dfs.short.circuit.shared.memory.watcher.interrupt.check.ms\":\"60000\",\"dfs.storage.policy.enabled\":\"true\",\"dfs.stream-buffer-size\":\"4096\",\"dfs.user.home.dir.prefix\":\"\\/user\",\"dfs.webhdfs.enabled\":\"true\",\"dfs.webhdfs.rest-csrf.browser-useragents-regex\":\"^Mozilla.*,^Opera.*\",\"dfs.webhdfs.rest-csrf.custom-header\":\"X-XSRF-HEADER\",\"dfs.webhdfs.rest-csrf.enabled\":\"false\",\"dfs.webhdfs.rest-csrf.methods-to-ignore\":\"GET,OPTIONS,HEAD,TRACE\",\"dfs.webhdfs.ugi.expire.after.access\":\"600000\",\"dfs.webhdfs.user.provider.user.pattern\":\"^[A-Za-z_][A-Za-z0-9._-]*[$]?$\",\"file.blocksize\":\"67108864\",\"file.bytes-per-checksum\":\"512\",\"file.client-write-packet-size\":\"65536\",\"file.replication\":\"1\",\"file.stream-buffer-size\":\"4096\",\"fs.AbstractFileSystem.file.impl\":\"org.apache.hadoop.fs.local.LocalFs\",\"fs.AbstractFileSystem.ftp.impl\":\"org.apache.hadoop.fs.ftp.FtpFs\",\"fs.AbstractFileSystem.har.impl\":\"org.apache.hadoop.fs.HarFs\",\"fs.AbstractFileSystem.hdfs.impl\":\"org.apache.hadoop.fs.Hdfs\",\"fs.AbstractFileSystem.swebhdfs.impl\":\"org.apache.hadoop.fs.SWebHdfs\",\"fs.AbstractFileSystem.viewfs.impl\":\"org.apache.hadoop.fs.viewfs.ViewFs\",\"fs.AbstractFileSystem.webhdfs.impl\":\"org.apache.hadoop.fs.WebHdfs\",\"fs.automatic.close\":\"false\",\"fs.client.resolve.remote.symlinks\":\"true\",\"fs.defaultFS\":\"hdfs:\\/\\/hdfs-cluster\",\"fs.df.interval\":\"60000\",\"fs.du.interval\":\"600000\",\"fs.ftp.host\":\"XXX.XXX.XXX.0\",\"fs.ftp.host.port\":\"21\",\"fs.har.impl.disable.cache\":\"true\",\"fs.permissions.umask-mode\":\"022\",\"fs.s3.block.size\":\"67108864\",\"fs.s3.buffer.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/s3\",\"fs.s3.maxRetries\":\"4\",\"fs.s3.sleepTimeSeconds\":\"10\",\"fs.s3a.attempts.maximum\":\"10\",\"fs.s3a.buffer.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/s3a\",\"fs.s3a.connection.establish.timeout\":\"5000\",\"fs.s3a.connection.maximum\":\"15\",\"fs.s3a.connection.ssl.enabled\":\"true\",\"fs.s3a.connection.timeout\":\"50000\",\"fs.s3a.fast.buffer.size\":\"1048576\",\"fs.s3a.fast.upload\":\"false\",\"fs.s3a.impl\":\"org.apache.hadoop.fs.s3a.S3AFileSystem\",\"fs.s3a.max.total.tasks\":\"1000\",\"fs.s3a.multipart.purge\":\"false\",\"fs.s3a.multipart.purge.age\":\"86400\",\"fs.s3a.multipart.size\":\"104857600\",\"fs.s3a.multipart.threshold\":\"2147483647\",\"fs.s3a.paging.maximum\":\"5000\",\"fs.s3a.threads.core\":\"15\",\"fs.s3a.threads.keepalivetime\":\"60\",\"fs.s3a.threads.max\":\"256\",\"fs.s3n.block.size\":\"67108864\",\"fs.s3n.multipart.copy.block.size\":\"5368709120\",\"fs.s3n.multipart.uploads.block.size\":\"67108864\",\"fs.s3n.multipart.uploads.enabled\":\"false\",\"fs.swift.impl\":\"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem\",\"fs.trash.checkpoint.interval\":\"0\",\"fs.trash.interval\":\"0\",\"ftp.blocksize\":\"67108864\",\"ftp.bytes-per-checksum\":\"512\",\"ftp.client-write-packet-size\":\"65536\",\"ftp.replication\":\"3\",\"ftp.stream-buffer-size\":\"4096\",\"ha.failover-controller.cli-check.rpc-timeout.ms\":\"20000\",\"ha.failover-controller.graceful-fence.connection.retries\":\"1\",\"ha.failover-controller.graceful-fence.rpc-timeout.ms\":\"5000\",\"ha.failover-controller.new-active.rpc-timeout.ms\":\"60000\",\"ha.health-monitor.check-interval.ms\":\"1000\",\"ha.health-monitor.connect-retry-interval.ms\":\"1000\",\"ha.health-monitor.rpc-timeout.ms\":\"45000\",\"ha.health-monitor.sleep-after-disconnect.ms\":\"1000\",\"ha.zookeeper.acl\":\"world:anyone:rwcda\",\"ha.zookeeper.parent-znode\":\"\\/hadoop-ha\",\"ha.zookeeper.quorum\":\"testvm001:2181,testvm002:2181,testvm003:2181\",\"ha.zookeeper.session-timeout.ms\":\"5000\",\"hadoop.common.configuration.version\":\"0.23.0\",\"hadoop.fuse.connection.timeout\":\"300\",\"hadoop.fuse.timer.period\":\"5\",\"hadoop.hdfs.configuration.version\":\"1\",\"hadoop.http.authentication.kerberos.keytab\":\"\\/home\\/ansible\\/hadoop.keytab\",\"hadoop.http.authentication.kerberos.principal\":\"HTTP\\/_HOST@LOCALHOST\",\"hadoop.http.authentication.signature.secret.file\":\"\\/home\\/ansible\\/hadoop-http-auth-signature-secret\",\"hadoop.http.authentication.simple.anonymous.allowed\":\"true\",\"hadoop.http.authentication.token.validity\":\"36000\",\"hadoop.http.authentication.type\":\"simple\",\"hadoop.http.cross-origin.allowed-headers\":\"X-Requested-With,Content-Type,Accept,Origin\",\"hadoop.http.cross-origin.allowed-methods\":\"GET,POST,HEAD\",\"hadoop.http.cross-origin.allowed-origins\":\"*\",\"hadoop.http.cross-origin.enabled\":\"false\",\"hadoop.http.cross-origin.max-age\":\"1800\",\"hadoop.http.filter.initializers\":\"org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.security.HttpCrossOriginFilterInitializer\",\"hadoop.http.staticuser.user\":\"dr.who\",\"hadoop.jetty.logs.serve.aliases\":\"true\",\"hadoop.kerberos.kinit.command\":\"kinit\",\"hadoop.proxyuser.hcat.groups\":\"*\",\"hadoop.proxyuser.hcat.hosts\":\"*\",\"hadoop.proxyuser.hdfs.groups\":\"*\",\"hadoop.proxyuser.hdfs.hosts\":\"*\",\"hadoop.proxyuser.httpfs.groups\":\"*\",\"hadoop.proxyuser.httpfs.hosts\":\"*\",\"hadoop.proxyuser.hue.groups\":\"*\",\"hadoop.proxyuser.hue.hosts\":\"*\",\"hadoop.registry.jaas.context\":\"Client\",\"hadoop.registry.rm.enabled\":\"false\",\"hadoop.registry.secure\":\"false\",\"hadoop.registry.system.acls\":\"sasl:yarn@, sasl:mapred@, sasl:hdfs@\",\"hadoop.registry.zk.connection.timeout.ms\":\"15000\",\"hadoop.registry.zk.quorum\":\"localhost:2181\",\"hadoop.registry.zk.retry.ceiling.ms\":\"60000\",\"hadoop.registry.zk.retry.interval.ms\":\"1000\",\"hadoop.registry.zk.retry.times\":\"5\",\"hadoop.registry.zk.root\":\"\\/registry\",\"hadoop.registry.zk.session.timeout.ms\":\"60000\",\"hadoop.rpc.protection\":\"authentication\",\"hadoop.rpc.socket.factory.class.default\":\"org.apache.hadoop.net.StandardSocketFactory\",\"hadoop.security.authentication\":\"simple\",\"hadoop.security.authorization\":\"false\",\"hadoop.security.crypto.buffer.size\":\"8192\",\"hadoop.security.crypto.cipher.suite\":\"AES\\/CTR\\/NoPadding\",\"hadoop.security.crypto.codec.classes.aes.ctr.nopadding\":\"org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec,org.apache.hadoop.crypto.JceAesCtrCryptoCodec\",\"hadoop.security.dns.log-slow-lookups.enabled\":\"false\",\"hadoop.security.dns.log-slow-lookups.threshold.ms\":\"1000\",\"hadoop.security.group.mapping\":\"org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback\",\"hadoop.security.group.mapping.ldap.directory.search.timeout\":\"10000\",\"hadoop.security.group.mapping.ldap.search.attr.group.name\":\"cn\",\"hadoop.security.group.mapping.ldap.search.attr.member\":\"member\",\"hadoop.security.group.mapping.ldap.search.filter.group\":\"(objectClass=group)\",\"hadoop.security.group.mapping.ldap.search.filter.user\":\"(&(objectClass=user)(sAMAccountName={0}))\",\"hadoop.security.group.mapping.ldap.ssl\":\"false\",\"hadoop.security.groups.cache.secs\":\"300\",\"hadoop.security.groups.cache.warn.after.ms\":\"5000\",\"hadoop.security.groups.negative-cache.secs\":\"30\",\"hadoop.security.instrumentation.requires.admin\":\"false\",\"hadoop.security.java.secure.random.algorithm\":\"SHA1PRNG\",\"hadoop.security.kms.client.authentication.retry-count\":\"1\",\"hadoop.security.kms.client.encrypted.key.cache.expiry\":\"43200000\",\"hadoop.security.kms.client.encrypted.key.cache.low-watermark\":\"0.3f\",\"hadoop.security.kms.client.encrypted.key.cache.num.refill.threads\":\"2\",\"hadoop.security.kms.client.encrypted.key.cache.size\":\"500\",\"hadoop.security.random.device.file.path\":\"\\/dev\\/urandom\",\"hadoop.security.uid.cache.secs\":\"14400\",\"hadoop.shell.safely.delete.limit.num.files\":\"100\",\"hadoop.ssl.client.conf\":\"ssl-client.xml\",\"hadoop.ssl.enabled\":\"false\",\"hadoop.ssl.enabled.protocols\":\"TLSv1\",\"hadoop.ssl.hostname.verifier\":\"DEFAULT\",\"hadoop.ssl.keystores.factory.class\":\"org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory\",\"hadoop.ssl.require.client.cert\":\"false\",\"hadoop.ssl.server.conf\":\"ssl-server.xml\",\"hadoop.tmp.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\",\"hadoop.user.group.static.mapping.overrides\":\"dr.who=;\",\"hadoop.util.hash.type\":\"murmur\",\"hadoop.work.around.non.threadsafe.getpwuid\":\"false\",\"io.compression.codec.bzip2.library\":\"system-native\",\"io.file.buffer.size\":\"4096\",\"io.map.index.interval\":\"128\",\"io.map.index.skip\":\"0\",\"io.mapfile.bloom.error.rate\":\"0.005\",\"io.mapfile.bloom.size\":\"1048576\",\"io.native.lib.available\":\"true\",\"io.seqfile.compress.blocksize\":\"1000000\",\"io.seqfile.lazydecompress\":\"true\",\"io.seqfile.local.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/io\\/local\",\"io.seqfile.sorter.recordlimit\":\"1000000\",\"io.serializations\":\"org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\",\"io.skip.checksum.errors\":\"false\",\"ipc.client.connect.max.retries\":\"10\",\"ipc.client.connect.max.retries.on.timeouts\":\"45\",\"ipc.client.connect.retry.interval\":\"1000\",\"ipc.client.connect.timeout\":\"20000\",\"ipc.client.connection.maxidletime\":\"10000\",\"ipc.client.fallback-to-simple-auth-allowed\":\"false\",\"ipc.client.idlethreshold\":\"4000\",\"ipc.client.kill.max\":\"10\",\"ipc.server.listen.queue.size\":\"128\",\"ipc.server.log.slow.rpc\":\"false\",\"ipc.server.max.connections\":\"0\",\"map.sort.class\":\"org.apache.hadoop.util.QuickSort\",\"mapred.child.java.opts\":\"-Xmx200m\",\"mapred.mapper.new-api\":\"true\",\"mapred.reducer.new-api\":\"true\",\"mapreduce.am.max-attempts\":\"2\",\"mapreduce.app-submission.cross-platform\":\"false\",\"mapreduce.client.completion.pollinterval\":\"5000\",\"mapreduce.client.genericoptionsparser.used\":\"true\",\"mapreduce.client.output.filter\":\"FAILED\",\"mapreduce.client.progressmonitor.pollinterval\":\"1000\",\"mapreduce.client.submit.file.replication\":\"10\",\"mapreduce.cluster.acls.enabled\":\"false\",\"mapreduce.cluster.local.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/local\",\"mapreduce.cluster.temp.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/temp\",\"mapreduce.fileoutputcommitter.algorithm.version\":\"1\",\"mapreduce.framework.name\":\"yarn-tez\",\"mapreduce.ifile.readahead\":\"true\",\"mapreduce.ifile.readahead.bytes\":\"4194304\",\"mapreduce.input.fileinputformat.inputdir\":\"hdfs:\\/\\/hdfs-cluster\\/user\\/ansible\\/QuasiMonteCarlo_1471570610739_682679022\\/in\",\"mapreduce.input.fileinputformat.list-status.num-threads\":\"1\",\"mapreduce.input.fileinputformat.split.minsize\":\"0\",\"mapreduce.input.lineinputformat.linespermap\":\"1\",\"mapreduce.job.acl-modify-job\":\" \",\"mapreduce.job.acl-view-job\":\" \",\"mapreduce.job.classloader\":\"false\",\"mapreduce.job.committer.setup.cleanup.needed\":\"true\",\"mapreduce.job.complete.cancel.delegation.tokens\":\"true\",\"mapreduce.job.counters.max\":\"120\",\"mapreduce.job.emit-timeline-data\":\"false\",\"mapreduce.job.end-notification.max.attempts\":\"5\",\"mapreduce.job.end-notification.max.retry.interval\":\"5000\",\"mapreduce.job.end-notification.retry.attempts\":\"0\",\"mapreduce.job.end-notification.retry.interval\":\"1000\",\"mapreduce.job.hdfs-servers\":\"hdfs:\\/\\/hdfs-cluster\",\"mapreduce.job.inputformat.class\":\"org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat\",\"mapreduce.job.jar\":\"\\/usr\\/hdp\\/XXX.XXX.XXX.0-258\\/hadoop-mapreduce\\/hadoop-mapreduce-examples-XXX.XXX.XXX.2.4.2.0-258.jar\",\"mapreduce.job.jvm.numtasks\":\"1\",\"mapreduce.job.map.class\":\"org.apache.hadoop.examples.QuasiMonteCarlo$QmcMapper\",\"mapreduce.job.map.output.collector.class\":\"org.apache.hadoop.mapred.MapTask$MapOutputBuffer\",\"mapreduce.job.maps\":\"2\",\"mapreduce.job.max.split.locations\":\"10\",\"mapreduce.job.maxtaskfailures.per.tracker\":\"3\",\"mapreduce.job.name\":\"QuasiMonteCarlo\",\"mapreduce.job.output.key.class\":\"org.apache.hadoop.io.BooleanWritable\",\"mapreduce.job.output.value.class\":\"org.apache.hadoop.io.LongWritable\",\"mapreduce.job.outputformat.class\":\"org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat\",\"mapreduce.job.queuename\":\"default\",\"mapreduce.job.reduce.class\":\"org.apache.hadoop.examples.QuasiMonteCarlo$QmcReducer\",\"mapreduce.job.reduce.shuffle.consumer.plugin.class\":\"org.apache.hadoop.mapreduce.task.reduce.Shuffle\",\"mapreduce.job.reduce.slowstart.completedmaps\":\"0.05\",\"mapreduce.job.reducer.preempt.delay.sec\":\"0\",\"mapreduce.job.reducer.unconditional-preempt.delay.sec\":\"300\",\"mapreduce.job.reduces\":\"1\",\"mapreduce.job.running.map.limit\":\"0\",\"mapreduce.job.running.reduce.limit\":\"0\",\"mapreduce.job.speculative.minimum-allowed-tasks\":\"10\",\"mapreduce.job.speculative.retry-after-no-speculate\":\"1000\",\"mapreduce.job.speculative.retry-after-speculate\":\"15000\",\"mapreduce.job.speculative.slowtaskthreshold\":\"1.0\",\"mapreduce.job.speculative.speculative-cap-running-tasks\":\"0.1\",\"mapreduce.job.speculative.speculative-cap-total-tasks\":\"0.01\",\"mapreduce.job.split.metainfo.maxsize\":\"10000000\",\"mapreduce.job.token.tracking.ids.enabled\":\"false\",\"mapreduce.job.ubertask.enable\":\"false\",\"mapreduce.job.ubertask.maxmaps\":\"9\",\"mapreduce.job.ubertask.maxreduces\":\"1\",\"mapreduce.job.userlog.retain.hours\":\"24\",\"mapreduce.jobhistory.address\":\"testvm003:10020\",\"mapreduce.jobhistory.admin.acl\":\"*\",\"mapreduce.jobhistory.admin.address\":\"XXX.XXX.XXX.0:10033\",\"mapreduce.jobhistory.cleaner.enable\":\"true\",\"mapreduce.jobhistory.cleaner.interval-ms\":\"86400000\",\"mapreduce.jobhistory.client.thread-count\":\"10\",\"mapreduce.jobhistory.datestring.cache.size\":\"200000\",\"mapreduce.jobhistory.done-dir\":\"\\/tmp\\/hadoop-yarn\\/staging\\/history\\/done\",\"mapreduce.jobhistory.http.policy\":\"HTTP_ONLY\",\"mapreduce.jobhistory.intermediate-done-dir\":\"\\/tmp\\/hadoop-yarn\\/staging\\/history\\/done_intermediate\",\"mapreduce.jobhistory.joblist.cache.size\":\"20000\",\"mapreduce.jobhistory.keytab\":\"\\/etc\\/security\\/keytab\\/jhs.service.keytab\",\"mapreduce.jobhistory.loadedjobs.cache.size\":\"5\",\"mapreduce.jobhistory.max-age-ms\":\"604800000\",\"mapreduce.jobhistory.minicluster.fixed.ports\":\"false\",\"mapreduce.jobhistory.move.interval-ms\":\"180000\",\"mapreduce.jobhistory.move.thread-count\":\"3\",\"mapreduce.jobhistory.principal\":\"jhs\\/_HOST@REALM.TLD\",\"mapreduce.jobhistory.recovery.enable\":\"false\",\"mapreduce.jobhistory.recovery.store.class\":\"org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService\",\"mapreduce.jobhistory.recovery.store.fs.uri\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/history\\/recoverystore\",\"mapreduce.jobhistory.recovery.store.leveldb.path\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/history\\/recoverystore\",\"mapreduce.jobhistory.webapp.address\":\"testvm003:19888\",\"mapreduce.jobhistory.webapp.rest-csrf.custom-header\":\"X-XSRF-Header\",\"mapreduce.jobhistory.webapp.rest-csrf.enabled\":\"false\",\"mapreduce.jobhistory.webapp.rest-csrf.methods-to-ignore\":\"GET,OPTIONS,HEAD\",\"mapreduce.jobtracker.address\":\"local\",\"mapreduce.jobtracker.expire.trackers.interval\":\"600000\",\"mapreduce.jobtracker.handler.count\":\"10\",\"mapreduce.jobtracker.heartbeats.in.second\":\"100\",\"mapreduce.jobtracker.http.address\":\"XXX.XXX.XXX.0:50030\",\"mapreduce.jobtracker.instrumentation\":\"org.apache.hadoop.mapred.JobTrackerMetricsInst\",\"mapreduce.jobtracker.jobhistory.block.size\":\"3145728\",\"mapreduce.jobtracker.jobhistory.lru.cache.size\":\"5\",\"mapreduce.jobtracker.jobhistory.task.numberprogresssplits\":\"12\",\"mapreduce.jobtracker.maxtasks.perjob\":\"-1\",\"mapreduce.jobtracker.persist.jobstatus.active\":\"true\",\"mapreduce.jobtracker.persist.jobstatus.dir\":\"\\/jobtracker\\/jobsInfo\",\"mapreduce.jobtracker.persist.jobstatus.hours\":\"1\",\"mapreduce.jobtracker.restart.recover\":\"false\",\"mapreduce.jobtracker.retiredjobs.cache.size\":\"1000\",\"mapreduce.jobtracker.staging.root.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/staging\",\"mapreduce.jobtracker.system.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/mapred\\/system\",\"mapreduce.jobtracker.taskcache.levels\":\"2\",\"mapreduce.jobtracker.taskscheduler\":\"org.apache.hadoop.mapred.JobQueueTaskScheduler\",\"mapreduce.jobtracker.tasktracker.maxblacklists\":\"4\",\"mapreduce.local.clientfactory.class.name\":\"org.apache.hadoop.mapred.LocalClientFactory\",\"mapreduce.map.cpu.vcores\":\"1\",\"mapreduce.map.java.opts\":\"-Xmx3840m\",\"mapreduce.map.log.level\":\"INFO\",\"mapreduce.map.maxattempts\":\"4\",\"mapreduce.map.memory.mb\":\"4096\",\"mapreduce.map.output.compress\":\"false\",\"mapreduce.map.output.compress.codec\":\"org.apache.hadoop.io.compress.DefaultCodec\",\"mapreduce.map.skip.maxrecords\":\"0\",\"mapreduce.map.skip.proc.count.autoincr\":\"true\",\"mapreduce.map.sort.spill.percent\":\"0.80\",\"mapreduce.map.speculative\":\"false\",\"mapreduce.output.fileoutputformat.compress\":\"false\",\"mapreduce.output.fileoutputformat.compress.codec\":\"org.apache.hadoop.io.compress.DefaultCodec\",\"mapreduce.output.fileoutputformat.compress.type\":\"RECORD\",\"mapreduce.output.fileoutputformat.outputdir\":\"hdfs:\\/\\/hdfs-cluster\\/user\\/ansible\\/QuasiMonteCarlo_1471570610739_682679022\\/out\",\"mapreduce.reduce.cpu.vcores\":\"1\",\"mapreduce.reduce.input.buffer.percent\":\"0.0\",\"mapreduce.reduce.java.opts\":\"-Xmx3840m\",\"mapreduce.reduce.log.level\":\"INFO\",\"mapreduce.reduce.markreset.buffer.percent\":\"0.0\",\"mapreduce.reduce.maxattempts\":\"4\",\"mapreduce.reduce.memory.mb\":\"4096\",\"mapreduce.reduce.merge.inmem.threshold\":\"1000\",\"mapreduce.reduce.shuffle.connect.timeout\":\"180000\",\"mapreduce.reduce.shuffle.fetch.retry.enabled\":\"true\",\"mapreduce.reduce.shuffle.fetch.retry.interval-ms\":\"1000\",\"mapreduce.reduce.shuffle.fetch.retry.timeout-ms\":\"30000\",\"mapreduce.reduce.shuffle.input.buffer.percent\":\"0.70\",\"mapreduce.reduce.shuffle.memory.limit.percent\":\"0.25\",\"mapreduce.reduce.shuffle.merge.percent\":\"0.66\",\"mapreduce.reduce.shuffle.parallelcopies\":\"5\",\"mapreduce.reduce.shuffle.read.timeout\":\"180000\",\"mapreduce.reduce.shuffle.retry-delay.max.ms\":\"60000\",\"mapreduce.reduce.skip.maxgroups\":\"0\",\"mapreduce.reduce.skip.proc.count.autoincr\":\"true\",\"mapreduce.reduce.speculative\":\"false\",\"mapreduce.shuffle.connection-keep-alive.enable\":\"false\",\"mapreduce.shuffle.connection-keep-alive.timeout\":\"5\",\"mapreduce.shuffle.max.connections\":\"0\",\"mapreduce.shuffle.max.threads\":\"0\",\"mapreduce.shuffle.port\":\"13562\",\"mapreduce.shuffle.ssl.enabled\":\"false\",\"mapreduce.shuffle.ssl.file.buffer.size\":\"65536\",\"mapreduce.shuffle.transfer.buffer.size\":\"131072\",\"mapreduce.task.combine.progress.records\":\"10000\",\"mapreduce.task.files.preserve.failedtasks\":\"false\",\"mapreduce.task.io.sort.factor\":\"10\",\"mapreduce.task.io.sort.mb\":\"100\",\"mapreduce.task.merge.progress.records\":\"10000\",\"mapreduce.task.profile\":\"false\",\"mapreduce.task.profile.map.params\":\"-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s\",\"mapreduce.task.profile.maps\":\"0-2\",\"mapreduce.task.profile.params\":\"-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s\",\"mapreduce.task.profile.reduce.params\":\"-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s\",\"mapreduce.task.profile.reduces\":\"0-2\",\"mapreduce.task.skip.start.attempts\":\"2\",\"mapreduce.task.timeout\":\"600000\",\"mapreduce.task.userlog.limit.kb\":\"0\",\"mapreduce.tasktracker.dns.interface\":\"default\",\"mapreduce.tasktracker.dns.nameserver\":\"default\",\"mapreduce.tasktracker.healthchecker.interval\":\"60000\",\"mapreduce.tasktracker.healthchecker.script.timeout\":\"600000\",\"mapreduce.tasktracker.http.address\":\"XXX.XXX.XXX.0:50060\",\"mapreduce.tasktracker.http.threads\":\"40\",\"mapreduce.tasktracker.indexcache.mb\":\"10\",\"mapreduce.tasktracker.instrumentation\":\"org.apache.hadoop.mapred.TaskTrackerMetricsInst\",\"mapreduce.tasktracker.local.dir.minspacekill\":\"0\",\"mapreduce.tasktracker.local.dir.minspacestart\":\"0\",\"mapreduce.tasktracker.map.tasks.maximum\":\"2\",\"mapreduce.tasktracker.outofband.heartbeat\":\"false\",\"mapreduce.tasktracker.reduce.tasks.maximum\":\"2\",\"mapreduce.tasktracker.report.address\":\"XXX.XXX.XXX.1:0\",\"mapreduce.tasktracker.taskcontroller\":\"org.apache.hadoop.mapred.DefaultTaskController\",\"mapreduce.tasktracker.taskmemorymanager.monitoringinterval\":\"5000\",\"mapreduce.tasktracker.tasks.sleeptimebeforesigkill\":\"5000\",\"net.topology.impl\":\"org.apache.hadoop.net.NetworkTopology\",\"net.topology.node.switch.mapping.impl\":\"org.apache.hadoop.net.ScriptBasedMapping\",\"net.topology.script.number.args\":\"100\",\"nfs.allow.insecure.ports\":\"true\",\"nfs.dump.dir\":\"\\/tmp\\/.hdfs-nfs\",\"nfs.exports.allowed.hosts\":\"* rw\",\"nfs.mountd.port\":\"4242\",\"nfs.rtmax\":\"1048576\",\"nfs.server.port\":\"2049\",\"nfs.wtmax\":\"1048576\",\"rpc.engine.org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB\":\"org.apache.hadoop.ipc.ProtobufRpcEngine\",\"rpc.metrics.quantile.enable\":\"false\",\"s3.blocksize\":\"67108864\",\"s3.bytes-per-checksum\":\"512\",\"s3.client-write-packet-size\":\"65536\",\"s3.replication\":\"3\",\"s3.stream-buffer-size\":\"4096\",\"s3native.blocksize\":\"67108864\",\"s3native.bytes-per-checksum\":\"512\",\"s3native.client-write-packet-size\":\"65536\",\"s3native.replication\":\"3\",\"s3native.stream-buffer-size\":\"4096\",\"tez.am.launch.cmd-opts\":\" -Xmx3840m \",\"tez.am.max.app.attempts\":\"2\",\"tez.am.maxtaskfailures.per.node\":\"3\",\"tez.am.mode.session\":\"false\",\"tez.am.resource.cpu.vcores\":\"1\",\"tez.am.resource.memory.mb\":\"4096\",\"tez.am.task.listener.thread-count\":\"30\",\"tez.counters.max\":\"120\",\"tez.lib.uris\":\"hdfs:\\/\\/hdfs-cluster\\/\\/apps\\/tez\\/tez.tar.gz\",\"tez.queue.name\":\"default\",\"tez.staging-dir\":\"\\/tmp\\/hadoop-yarn\\/staging\\/ansible\\/.staging\\/job_1471559811619_0002\",\"tfile.fs.input.buffer.size\":\"262144\",\"tfile.fs.output.buffer.size\":\"262144\",\"tfile.io.chunk.size\":\"1048576\",\"yarn.acl.enable\":\"false\",\"yarn.admin.acl\":\"*\",\"yarn.am.liveness-monitor.expiry-interval-ms\":\"600000\",\"yarn.app.mapreduce.am.command-opts\":\"-Xmx3840m\",\"yarn.app.mapreduce.am.container.log.backups\":\"0\",\"yarn.app.mapreduce.am.container.log.limit.kb\":\"0\",\"yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size\":\"10\",\"yarn.app.mapreduce.am.hard-kill-timeout-ms\":\"10000\",\"yarn.app.mapreduce.am.job.committer.cancel-timeout\":\"60000\",\"yarn.app.mapreduce.am.job.committer.commit-window\":\"10000\",\"yarn.app.mapreduce.am.job.task.listener.thread-count\":\"30\",\"yarn.app.mapreduce.am.resource.cpu-vcores\":\"1\",\"yarn.app.mapreduce.am.resource.mb\":\"4096\",\"yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms\":\"1000\",\"yarn.app.mapreduce.am.staging-dir\":\"\\/tmp\\/hadoop-yarn\\/staging\",\"yarn.app.mapreduce.client-am.ipc.max-retries\":\"3\",\"yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts\":\"3\",\"yarn.app.mapreduce.client.job.max-retries\":\"0\",\"yarn.app.mapreduce.client.job.retry-interval\":\"2000\",\"yarn.app.mapreduce.client.max-retries\":\"3\",\"yarn.app.mapreduce.shuffle.log.backups\":\"0\",\"yarn.app.mapreduce.shuffle.log.limit.kb\":\"0\",\"yarn.app.mapreduce.shuffle.log.separate\":\"true\",\"yarn.app.mapreduce.task.container.log.backups\":\"0\",\"yarn.application.classpath\":\"\\n     \\/usr\\/hdp\\/current\\/hadoop-client\\/*,\\/usr\\/hdp\\/current\\/hadoop-client\\/lib\\/*,\\n     \\/usr\\/hdp\\/current\\/hadoop-hdfs-client\\/*,\\/usr\\/hdp\\/current\\/hadoop-hdfs-client\\/lib\\/*,\\n     \\/usr\\/hdp\\/current\\/hadoop-mapreduce-client\\/*,\\/usr\\/hdp\\/current\\/hadoop-mapreduce-client\\/lib\\/*,\\n     \\/usr\\/hdp\\/current\\/hadoop-yarn-client\\/*,\\/usr\\/hdp\\/current\\/hadoop-yarn-client\\/lib\\/*,\\n     \\/etc\\/tez\\/conf,\\n     \\/usr\\/hdp\\/current\\/tez-client\\/*,user\\/hdp\\/current\\/tez-client\\/lib\\/*\\n  \",\"yarn.client.application-client-protocol.poll-interval-ms\":\"200\",\"yarn.client.failover-proxy-provider\":\"org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider\",\"yarn.client.failover-retries\":\"0\",\"yarn.client.failover-retries-on-socket-timeouts\":\"0\",\"yarn.client.max-cached-nodemanagers-proxies\":\"0\",\"yarn.client.nodemanager-client-async.thread-pool-max-size\":\"500\",\"yarn.client.nodemanager-connect.max-wait-ms\":\"180000\",\"yarn.client.nodemanager-connect.retry-interval-ms\":\"10000\",\"yarn.dispatcher.drain-events.timeout\":\"300000\",\"yarn.dispatcher.exit-on-error\":\"true\",\"yarn.fail-fast\":\"false\",\"yarn.http.policy\":\"HTTP_ONLY\",\"yarn.ipc.rpc.class\":\"org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC\",\"yarn.log-aggregation-enable\":\"true\",\"yarn.log-aggregation.retain-check-interval-seconds\":\"-1\",\"yarn.log-aggregation.retain-seconds\":\"-1\",\"yarn.nm.liveness-monitor.expiry-interval-ms\":\"600000\",\"yarn.node-labels.fs-store.impl.class\":\"org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore\",\"yarn.nodemanager.address\":\"XXX.XXX.XXX.0:45454\",\"yarn.nodemanager.admin-env\":\"MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX\",\"yarn.nodemanager.aux-services\":\"mapreduce_shuffle\",\"yarn.nodemanager.aux-services.mapreduce_shuffle.class\":\"org.apache.hadoop.mapred.ShuffleHandler\",\"yarn.nodemanager.container-executor.class\":\"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor\",\"yarn.nodemanager.container-manager.thread-count\":\"20\",\"yarn.nodemanager.container-metrics.unregister-delay-ms\":\"10000\",\"yarn.nodemanager.container-monitor.interval-ms\":\"3000\",\"yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled\":\"false\",\"yarn.nodemanager.delete.debug-delay-sec\":\"3600\",\"yarn.nodemanager.delete.thread-count\":\"4\",\"yarn.nodemanager.disk-health-checker.interval-ms\":\"120000\",\"yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage\":\"90.0\",\"yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb\":\"0\",\"yarn.nodemanager.disk-health-checker.min-healthy-disks\":\"0.25\",\"yarn.nodemanager.docker-container-executor.exec-name\":\"\\/usr\\/bin\\/docker\",\"yarn.nodemanager.env-whitelist\":\"JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,HADOOP_YARN_HOME\",\"yarn.nodemanager.health-checker.interval-ms\":\"600000\",\"yarn.nodemanager.health-checker.script.timeout-ms\":\"1200000\",\"yarn.nodemanager.hostname\":\"XXX.XXX.XXX.0\",\"yarn.nodemanager.keytab\":\"\\/etc\\/krb5.keytab\",\"yarn.nodemanager.linux-container-executor.cgroups.hierarchy\":\"\\/hadoop-yarn\",\"yarn.nodemanager.linux-container-executor.cgroups.mount\":\"false\",\"yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage\":\"false\",\"yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users\":\"true\",\"yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user\":\"nobody\",\"yarn.nodemanager.linux-container-executor.nonsecure-mode.user-pattern\":\"^[_.A-Za-z0-9][-@_.A-Za-z0-9]{0,255}?[$]?$\",\"yarn.nodemanager.linux-container-executor.resources-handler.class\":\"org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler\",\"yarn.nodemanager.local-cache.max-files-per-directory\":\"8192\",\"yarn.nodemanager.local-dirs\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/nm-local-dir\",\"yarn.nodemanager.localizer.address\":\"XXX.XXX.XXX.0:8040\",\"yarn.nodemanager.localizer.cache.cleanup.interval-ms\":\"600000\",\"yarn.nodemanager.localizer.cache.target-size-mb\":\"10240\",\"yarn.nodemanager.localizer.client.thread-count\":\"5\",\"yarn.nodemanager.localizer.fetch.thread-count\":\"4\",\"yarn.nodemanager.log-aggregation.compression-type\":\"none\",\"yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds\":\"-1\",\"yarn.nodemanager.log-container-debug-info.enabled\":\"true\",\"yarn.nodemanager.log-dirs\":\"\\/var\\/log\\/hadoop-yarn\\/userlogs\",\"yarn.nodemanager.log.retain-seconds\":\"10800\",\"yarn.nodemanager.pmem-check-enabled\":\"true\",\"yarn.nodemanager.process-kill-wait.ms\":\"2000\",\"yarn.nodemanager.recovery.dir\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/yarn-nm-recovery\",\"yarn.nodemanager.recovery.enabled\":\"true\",\"yarn.nodemanager.remote-app-log-dir\":\"hdfs:\\/\\/hdfs-cluster\\/var\\/log\\/yarn\",\"yarn.nodemanager.remote-app-log-dir-suffix\":\"logs\",\"yarn.nodemanager.resource.cpu-vcores\":\"8\",\"yarn.nodemanager.resource.memory-mb\":\"8192\",\"yarn.nodemanager.resource.percentage-physical-cpu-limit\":\"100\",\"yarn.nodemanager.resourcemanager.minimum.version\":\"NONE\",\"yarn.nodemanager.sleep-delay-before-sigkill.ms\":\"250\",\"yarn.nodemanager.vmem-check-enabled\":\"true\",\"yarn.nodemanager.vmem-pmem-ratio\":\"2.1\",\"yarn.nodemanager.webapp.address\":\"XXX.XXX.XXX.0:8042\",\"yarn.nodemanager.webapp.cross-origin.enabled\":\"false\",\"yarn.nodemanager.webapp.rest-csrf.custom-header\":\"X-XSRF-Header\",\"yarn.nodemanager.webapp.rest-csrf.enabled\":\"false\",\"yarn.nodemanager.webapp.rest-csrf.methods-to-ignore\":\"GET,OPTIONS,HEAD\",\"yarn.nodemanager.windows-container.cpu-limit.enabled\":\"false\",\"yarn.nodemanager.windows-container.memory-limit.enabled\":\"false\",\"yarn.resourcemanager.address\":\"XXX.XXX.XXX.0:8032\",\"yarn.resourcemanager.address.testvm001\":\"testvm001:8032\",\"yarn.resourcemanager.address.testvm002\":\"testvm002:8032\",\"yarn.resourcemanager.admin.address\":\"XXX.XXX.XXX.0:8033\",\"yarn.resourcemanager.admin.address.testvm001\":\"testvm001:8033\",\"yarn.resourcemanager.admin.address.testvm002\":\"testvm002:8033\",\"yarn.resourcemanager.admin.client.thread-count\":\"1\",\"yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs\":\"86400\",\"yarn.resourcemanager.am.max-attempts\":\"2\",\"yarn.resourcemanager.amlauncher.thread-count\":\"50\",\"yarn.resourcemanager.client.thread-count\":\"50\",\"yarn.resourcemanager.cluster-id\":\"yarn-cluster\",\"yarn.resourcemanager.configuration.provider-class\":\"org.apache.hadoop.yarn.LocalConfigurationProvider\",\"yarn.resourcemanager.connect.max-wait.ms\":\"900000\",\"yarn.resourcemanager.connect.retry-interval.ms\":\"30000\",\"yarn.resourcemanager.container-tokens.master-key-rolling-interval-secs\":\"86400\",\"yarn.resourcemanager.container.liveness-monitor.interval-ms\":\"600000\",\"yarn.resourcemanager.delayed.delegation-token.removal-interval-ms\":\"30000\",\"yarn.resourcemanager.fail-fast\":\"false\",\"yarn.resourcemanager.fs.state-store.num-retries\":\"0\",\"yarn.resourcemanager.fs.state-store.retry-interval-ms\":\"1000\",\"yarn.resourcemanager.fs.state-store.retry-policy-spec\":\"2000, 500\",\"yarn.resourcemanager.fs.state-store.uri\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/yarn\\/system\\/rmstore\",\"yarn.resourcemanager.ha.automatic-failover.embedded\":\"true\",\"yarn.resourcemanager.ha.automatic-failover.enabled\":\"true\",\"yarn.resourcemanager.ha.automatic-failover.zk-base-path\":\"\\/yarn-leader-election\",\"yarn.resourcemanager.ha.enabled\":\"true\",\"yarn.resourcemanager.ha.rm-ids\":\"testvm001,testvm002\",\"yarn.resourcemanager.hostname\":\"XXX.XXX.XXX.0\",\"yarn.resourcemanager.keytab\":\"\\/etc\\/krb5.keytab\",\"yarn.resourcemanager.leveldb-state-store.path\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/yarn\\/system\\/rmstore\",\"yarn.resourcemanager.max-completed-applications\":\"10000\",\"yarn.resourcemanager.max-log-aggregation-diagnostics-in-memory\":\"10\",\"yarn.resourcemanager.nodemanager-connect-retries\":\"10\",\"yarn.resourcemanager.nodemanager.minimum.version\":\"NONE\",\"yarn.resourcemanager.nodemanagers.heartbeat-interval-ms\":\"1000\",\"yarn.resourcemanager.nodes.exclude-path\":\"\\/etc\\/hadoop\\/conf\\/hosts.exclude\",\"yarn.resourcemanager.nodes.include-path\":\"\\/etc\\/hadoop\\/conf\\/hosts.list\",\"yarn.resourcemanager.proxy-user-privileges.enabled\":\"false\",\"yarn.resourcemanager.recovery.enabled\":\"true\",\"yarn.resourcemanager.resource-tracker.address\":\"XXX.XXX.XXX.0:8031\",\"yarn.resourcemanager.resource-tracker.address.testvm001\":\"testvm001:8031\",\"yarn.resourcemanager.resource-tracker.address.testvm002\":\"testvm002:8031\",\"yarn.resourcemanager.resource-tracker.client.thread-count\":\"50\",\"yarn.resourcemanager.scheduler.address\":\"XXX.XXX.XXX.0:8030\",\"yarn.resourcemanager.scheduler.address.testvm001\":\"testvm001:8030\",\"yarn.resourcemanager.scheduler.address.testvm002\":\"testvm002:8030\",\"yarn.resourcemanager.scheduler.class\":\"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler\",\"yarn.resourcemanager.scheduler.client.thread-count\":\"50\",\"yarn.resourcemanager.scheduler.monitor.enable\":\"false\",\"yarn.resourcemanager.scheduler.monitor.policies\":\"org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy\",\"yarn.resourcemanager.state-store.max-completed-applications\":\"10000\",\"yarn.resourcemanager.store.class\":\"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore\",\"yarn.resourcemanager.system-metrics-publisher.dispatcher.pool-size\":\"10\",\"yarn.resourcemanager.system-metrics-publisher.enabled\":\"true\",\"yarn.resourcemanager.webapp.address\":\"XXX.XXX.XXX.0:8088\",\"yarn.resourcemanager.webapp.address.testvm001\":\"testvm001:8088\",\"yarn.resourcemanager.webapp.address.testvm002\":\"testvm002:8088\",\"yarn.resourcemanager.webapp.cross-origin.enabled\":\"true\",\"yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled\":\"true\",\"yarn.resourcemanager.webapp.https.address\":\"XXX.XXX.XXX.0:8090\",\"yarn.resourcemanager.webapp.rest-csrf.custom-header\":\"X-XSRF-Header\",\"yarn.resourcemanager.webapp.rest-csrf.enabled\":\"false\",\"yarn.resourcemanager.webapp.rest-csrf.methods-to-ignore\":\"GET,OPTIONS,HEAD\",\"yarn.resourcemanager.work-preserving-recovery.enabled\":\"true\",\"yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms\":\"10000\",\"yarn.resourcemanager.zk-acl\":\"world:anyone:rwcda\",\"yarn.resourcemanager.zk-address\":\"testvm001:2181,testvm002:2181,testvm003:2181\",\"yarn.resourcemanager.zk-num-retries\":\"1000\",\"yarn.resourcemanager.zk-retry-interval-ms\":\"1000\",\"yarn.resourcemanager.zk-state-store.parent-path\":\"\\/rmstore\",\"yarn.resourcemanager.zk-timeout-ms\":\"10000\",\"yarn.scheduler.maximum-allocation-mb\":\"8192\",\"yarn.scheduler.maximum-allocation-vcores\":\"32\",\"yarn.scheduler.minimum-allocation-mb\":\"1024\",\"yarn.scheduler.minimum-allocation-vcores\":\"1\",\"yarn.sharedcache.admin.address\":\"XXX.XXX.XXX.0:8047\",\"yarn.sharedcache.admin.thread-count\":\"1\",\"yarn.sharedcache.app-checker.class\":\"org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker\",\"yarn.sharedcache.checksum.algo.impl\":\"org.apache.hadoop.yarn.sharedcache.ChecksumSHA256Impl\",\"yarn.sharedcache.cleaner.initial-delay-mins\":\"10\",\"yarn.sharedcache.cleaner.period-mins\":\"1440\",\"yarn.sharedcache.cleaner.resource-sleep-ms\":\"0\",\"yarn.sharedcache.client-server.address\":\"XXX.XXX.XXX.0:8045\",\"yarn.sharedcache.client-server.thread-count\":\"50\",\"yarn.sharedcache.enabled\":\"false\",\"yarn.sharedcache.nested-level\":\"3\",\"yarn.sharedcache.nm.uploader.replication.factor\":\"10\",\"yarn.sharedcache.nm.uploader.thread-count\":\"20\",\"yarn.sharedcache.root-dir\":\"\\/sharedcache\",\"yarn.sharedcache.store.class\":\"org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore\",\"yarn.sharedcache.store.in-memory.check-period-mins\":\"720\",\"yarn.sharedcache.store.in-memory.initial-delay-mins\":\"10\",\"yarn.sharedcache.store.in-memory.staleness-period-mins\":\"10080\",\"yarn.sharedcache.uploader.server.address\":\"XXX.XXX.XXX.0:8046\",\"yarn.sharedcache.uploader.server.thread-count\":\"50\",\"yarn.sharedcache.webapp.address\":\"XXX.XXX.XXX.0:8788\",\"yarn.timeline-service.address\":\"testvm003:10200\",\"yarn.timeline-service.client.best-effort\":\"false\",\"yarn.timeline-service.client.max-retries\":\"30\",\"yarn.timeline-service.client.retry-interval-ms\":\"1000\",\"yarn.timeline-service.enabled\":\"true\",\"yarn.timeline-service.generic-application-history.max-applications\":\"10000\",\"yarn.timeline-service.generic-application-history.save-non-am-container-meta-info\":\"false\",\"yarn.timeline-service.handler-thread-count\":\"10\",\"yarn.timeline-service.hostname\":\"testvm003\",\"yarn.timeline-service.http-authentication.simple.anonymous.allowed\":\"true\",\"yarn.timeline-service.http-authentication.type\":\"simple\",\"yarn.timeline-service.http-cross-origin.enabled\":\"true\",\"yarn.timeline-service.keytab\":\"\\/etc\\/krb5.keytab\",\"yarn.timeline-service.leveldb-state-store.path\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/yarn\\/timeline\",\"yarn.timeline-service.leveldb-timeline-store.path\":\"\\/hadoop\\/tmp\\/hadoop-ansible\\/yarn\\/timeline\",\"yarn.timeline-service.leveldb-timeline-store.read-cache-size\":\"104857600\",\"yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size\":\"10000\",\"yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size\":\"10000\",\"yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms\":\"300000\",\"yarn.timeline-service.recovery.enabled\":\"false\",\"yarn.timeline-service.state-store-class\":\"org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore\",\"yarn.timeline-service.store-class\":\"org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore\",\"yarn.timeline-service.ttl-enable\":\"true\",\"yarn.timeline-service.ttl-ms\":\"604800000\",\"yarn.timeline-service.webapp.address\":\"testvm003:8188\",\"yarn.timeline-service.webapp.https.address\":\"testvm003:8190\",\"yarn.timeline-service.webapp.rest-csrf.custom-header\":\"X-XSRF-Header\",\"yarn.timeline-service.webapp.rest-csrf.enabled\":\"false\",\"yarn.timeline-service.webapp.rest-csrf.methods-to-ignore\":\"GET,OPTIONS,HEAD\"}}}\u0001\n",
      "{\"entity\":\"tez_appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\",\"relatedEntities\":[{\"entity\":\"application_1471559811619_0002\",\"entitytype\":\"applicationId\"},{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"applicationAttemptId\"}],\"events\":[{\"ts\":1471570622790,\"eventtype\":\"AM_LAUNCHED\"}],\"otherinfo\":{\"appSubmitTime\":1471570620028}}\u0001\n",
      "{\"entity\":\"tez_appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\",\"relatedEntities\":[{\"entity\":\"application_1471559811619_0002\",\"entitytype\":\"applicationId\"},{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"applicationAttemptId\"}],\"events\":[{\"ts\":1471570624855,\"eventtype\":\"AM_STARTED\"}]}\u0001\n",
      "{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\",\"relatedEntities\":[{\"entity\":\"tez_application_1471559811619_0002\",\"entitytype\":\"TEZ_APPLICATION\"},{\"entity\":\"tez_appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"application_1471559811619_0002\",\"entitytype\":\"applicationId\"},{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"applicationAttemptId\"},{\"entity\":\"ansible\",\"entitytype\":\"user\"}],\"primaryfilters\":{\"dagName\":\"QuasiMonteCarlo\"},\"events\":[{\"ts\":1471570624921,\"eventtype\":\"DAG_SUBMITTED\"}],\"otherinfo\":{\"dagPlan\":{\"dagName\":\"QuasiMonteCarlo\",\"version\":2,\"vertices\":[{\"vertexName\":\"initialmap\",\"processorClass\":\"org.apache.tez.mapreduce.processor.map.MapProcessor\",\"outEdgeIds\":[\"1738964628\"],\"additionalInputs\":[{\"name\":\"MRInput\",\"class\":\"org.apache.tez.mapreduce.input.MRInputLegacy\"}]},{\"vertexName\":\"finalreduce\",\"processorClass\":\"org.apache.tez.mapreduce.processor.reduce.ReduceProcessor\",\"inEdgeIds\":[\"1738964628\"],\"additionalOutputs\":[{\"name\":\"MROutput\",\"class\":\"org.apache.tez.mapreduce.output.MROutputLegacy\",\"initializer\":\"org.apache.tez.mapreduce.committer.MROutputCommitter\"}],\"vertexManagerPluginClass\":\"org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager\"}],\"edges\":[{\"edgeId\":\"1738964628\",\"inputVertexName\":\"initialmap\",\"outputVertexName\":\"finalreduce\",\"dataMovementType\":\"SCATTER_GATHER\",\"dataSourceType\":\"PERSISTED\",\"schedulingType\":\"SEQUENTIAL\",\"edgeSourceClass\":\"org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput\",\"edgeDestinationClass\":\"org.apache.tez.runtime.library.input.OrderedGroupedInputLegacy\"}]}}}\u0001\n",
      "{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\",\"events\":[{\"ts\":1471570625516,\"eventtype\":\"DAG_INITIALIZED\"}],\"otherinfo\":{\"vertexNameIdMapping\":{\"finalreduce\":\"vertex_1471559811619_0002_1_01\",\"initialmap\":\"vertex_1471559811619_0002_1_00\"}}}\u0001\n",
      "{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\",\"events\":[{\"ts\":1471570625740,\"eventtype\":\"DAG_STARTED\"}]}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\",\"relatedEntities\":[{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\"}],\"events\":[{\"ts\":1471570625789,\"eventtype\":\"VERTEX_INITIALIZED\"}],\"otherinfo\":{\"vertexName\":\"initialmap\",\"initRequestedTime\":1471570625748,\"initTime\":1471570625789,\"numTasks\":10,\"processorClassName\":\"org.apache.tez.mapreduce.processor.map.MapProcessor\"}}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\",\"relatedEntities\":[{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\"}],\"events\":[{\"ts\":1471570625792,\"eventtype\":\"VERTEX_STARTED\"}],\"otherinfo\":{\"startRequestedTime\":1471570625792,\"startTime\":1471570625792}}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_01\",\"entitytype\":\"TEZ_VERTEX_ID\",\"relatedEntities\":[{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\"}],\"events\":[{\"ts\":1471570625973,\"eventtype\":\"VERTEX_INITIALIZED\"}],\"otherinfo\":{\"vertexName\":\"finalreduce\",\"initRequestedTime\":1471570625812,\"initTime\":1471570625973,\"numTasks\":1,\"processorClassName\":\"org.apache.tez.mapreduce.processor.reduce.ReduceProcessor\"}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000000\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570625997,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570625997,\"scheduledTime\":1471570625997}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000001\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570625999,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570625999,\"scheduledTime\":1471570625999}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000002\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626003,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626003,\"scheduledTime\":1471570626003}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000003\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626003,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626003,\"scheduledTime\":1471570626003}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000004\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626004,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626004,\"scheduledTime\":1471570626004}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000005\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626007,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626007,\"scheduledTime\":1471570626007}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000006\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626007,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626007,\"scheduledTime\":1471570626007}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000007\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626008,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626008,\"scheduledTime\":1471570626008}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000008\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626009,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626009,\"scheduledTime\":1471570626009}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000009\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570626009,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570626009,\"scheduledTime\":1471570626009}}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_01\",\"entitytype\":\"TEZ_VERTEX_ID\",\"relatedEntities\":[{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\"}],\"events\":[{\"ts\":1471570626010,\"eventtype\":\"VERTEX_STARTED\"}],\"otherinfo\":{\"startRequestedTime\":1471570625977,\"startTime\":1471570626010}}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570628178,\"eventtype\":\"CONTAINER_LAUNCHED\"}]}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570628178,\"eventtype\":\"CONTAINER_LAUNCHED\"}]}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570628181,\"eventtype\":\"CONTAINER_LAUNCHED\"}]}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000000_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm007:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000000\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570630596,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm007:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000002\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000001_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm005:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000001\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570631097,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm005:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000003\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000002_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm006:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000002\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570631509,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm006:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000004\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000000_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570632615,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570625996,\"allocationTime\":1471570628065,\"startTime\":1471570630596,\"endTime\":1471570632615,\"timeTaken\":2019,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1849},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":250},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":2520},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":290979840},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":290979840},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000000\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570632628,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570630596,\"endTime\":1471570632628,\"timeTaken\":2032,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1849},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":250},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":2520},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":290979840},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":290979840},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000000_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000003_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm007:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000003\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570632642,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm007:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000002\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000003_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570633747,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626003,\"allocationTime\":1471570632633,\"startTime\":1471570632642,\"endTime\":1471570633747,\"timeTaken\":1105,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1411},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":850},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":396361728},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":396361728},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000003\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570633748,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570632642,\"endTime\":1471570633748,\"timeTaken\":1106,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1411},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":850},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":396361728},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":396361728},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000003_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000004_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm007:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000004\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570633850,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm007:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000002\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000001_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570633892,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570625999,\"allocationTime\":1471570628066,\"startTime\":1471570631097,\"endTime\":1471570633892,\"timeTaken\":2795,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1703},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":189},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3680},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":254803968},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":254803968},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000001\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570633896,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570631097,\"endTime\":1471570633896,\"timeTaken\":2799,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1703},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":189},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3680},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":254803968},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":254803968},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000001_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000005_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm005:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000005\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570633914,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm005:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000003\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000005_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570634059,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626006,\"allocationTime\":1471570633900,\"startTime\":1471570633914,\"endTime\":1471570634059,\"timeTaken\":145,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1119},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":170},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":360185856},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":360185856},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000005\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570634062,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570633914,\"endTime\":1471570634062,\"timeTaken\":148,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1119},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":170},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":360185856},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":360185856},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000005_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000002_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570634089,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626003,\"allocationTime\":1471570628067,\"startTime\":1471570631509,\"endTime\":1471570634089,\"timeTaken\":2580,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1557},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":208},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3400},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":290455552},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":290455552},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000002\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570634090,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570631509,\"endTime\":1471570634090,\"timeTaken\":2581,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1557},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":208},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3400},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":290455552},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":290455552},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000002_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000008_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm006:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000008\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570634103,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm006:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000004\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000006_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm005:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000006\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570634163,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm005:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000003\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000008_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570634260,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626009,\"allocationTime\":1471570634094,\"startTime\":1471570634103,\"endTime\":1471570634260,\"timeTaken\":157,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":681},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":200},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":395837440},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":395837440},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000008\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570634264,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570634103,\"endTime\":1471570634264,\"timeTaken\":161,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":681},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":200},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":395837440},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":395837440},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000008_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000006_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570634318,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626007,\"allocationTime\":1471570634064,\"startTime\":1471570634163,\"endTime\":1471570634318,\"timeTaken\":155,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":973},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":170},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":465567744},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":465567744},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000006\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570634321,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570634163,\"endTime\":1471570634321,\"timeTaken\":158,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":973},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":170},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":465567744},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":465567744},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000006_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000009_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm006:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000009\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570634367,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm006:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000004\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000007_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm005:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_00_000007\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570634425,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm005:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000003\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000007_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570634598,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626008,\"allocationTime\":1471570634327,\"startTime\":1471570634425,\"endTime\":1471570634598,\"timeTaken\":173,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":827},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":260},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":570949632},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":570949632},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000007\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570634602,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570634425,\"endTime\":1471570634602,\"timeTaken\":177,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":827},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":260},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":570949632},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4626919424},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":570949632},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000007_0\"}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_01_000000\",\"entitytype\":\"TEZ_TASK_ID\",\"relatedEntities\":[{\"entity\":\"vertex_1471559811619_0002_1_01\",\"entitytype\":\"TEZ_VERTEX_ID\"}],\"events\":[{\"ts\":1471570634611,\"eventtype\":\"TASK_STARTED\"}],\"otherinfo\":{\"startTime\":1471570634611,\"scheduledTime\":1471570634611}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_01_000000_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"relatedEntities\":[{\"entity\":\"testvm005:45454\",\"entitytype\":\"nodeId\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"},{\"entity\":\"task_1471559811619_0002_1_01_000000\",\"entitytype\":\"TEZ_TASK_ID\"}],\"events\":[{\"ts\":1471570634704,\"eventtype\":\"TASK_ATTEMPT_STARTED\"}],\"otherinfo\":{\"inProgressLogsURL\":\"testvm005:8042\\/node\\/containerlogs\\/container_e01_1471559811619_0002_01_000003\\/ansible\",\"completedLogsURL\":\"\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000009_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570638002,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626009,\"allocationTime\":1471570634278,\"startTime\":1471570634367,\"endTime\":1471570638002,\"timeTaken\":3635,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":535},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3200},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":501219328},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":501219328},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000009\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570638003,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570634367,\"endTime\":1471570638003,\"timeTaken\":3636,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":535},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":3200},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":501219328},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4624896000},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":501219328},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000009_0\"}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_00_000004_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570642939,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570626004,\"allocationTime\":1471570633754,\"startTime\":1471570633850,\"endTime\":1471570642939,\"timeTaken\":9089,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1265},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":6640},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":501743616},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":501743616},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_00_000004\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570642941,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570633850,\"endTime\":1471570642941,\"timeTaken\":9091,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":1}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":1265},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":64},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":118},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":2}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":146},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":2},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":6640},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":501743616},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4630061056},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":501743616},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":1},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":2},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":18},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":28},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":32},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":1}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_00_000004_0\"}}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_00\",\"entitytype\":\"TEZ_VERTEX_ID\",\"events\":[{\"ts\":1471570642948,\"eventtype\":\"VERTEX_FINISHED\"}],\"otherinfo\":{\"endTime\":1471570642948,\"timeTaken\":17156,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":10}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":11920},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":640},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":1180},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":20}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":1460},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":20},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":647},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":21090},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":4028104704},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":46272548864},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":4028104704},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":10},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":180},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":280},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":320},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":10}]}]},\"stats\":{\"firstTaskStartTime\":1471570630596,\"firstTasksToStart\":[\"task_1471559811619_0002_1_00_000000\"],\"lastTaskFinishTime\":1471570642939,\"lastTasksToFinish\":[\"task_1471559811619_0002_1_00_000004\"],\"minTaskDuration\":145,\"maxTaskDuration\":9089,\"avgTaskDuration\":2185.3,\"shortestDurationTasks\":[\"task_1471559811619_0002_1_00_000005\"],\"longestDurationTasks\":[\"task_1471559811619_0002_1_00_000004\"]},\"numFailedTasks\":0,\"numSucceededTasks\":10,\"numKilledTaskAttempts\":0,\"numKilledTasks\":0,\"numFailedTaskAttempts\":0,\"numCompletedTasks\":10}}\u0001\n",
      "{\"entity\":\"attempt_1471559811619_0002_1_01_000000_0\",\"entitytype\":\"TEZ_TASK_ATTEMPT_ID\",\"events\":[{\"ts\":1471570643474,\"eventtype\":\"TASK_ATTEMPT_FINISHED\"}],\"otherinfo\":{\"creationTime\":1471570634611,\"allocationTime\":1471570634617,\"startTime\":1471570634704,\"endTime\":1471570643474,\"timeTaken\":8770,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":382},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"HDFS_BYTES_WRITTEN\",\"counterValue\":215},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":4},{\"counterName\":\"HDFS_WRITE_OPS\",\"counterValue\":3}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"REDUCE_INPUT_GROUPS\",\"counterValue\":2},{\"counterName\":\"REDUCE_INPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":20},{\"counterName\":\"NUM_SHUFFLED_INPUTS\",\"counterValue\":10},{\"counterName\":\"MERGED_MAP_OUTPUTS\",\"counterValue\":10},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":179},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":1410},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":596115456},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4638515200},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":596115456},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_READ\",\"counterValue\":254},{\"counterName\":\"SHUFFLE_BYTES\",\"counterValue\":320},{\"counterName\":\"SHUFFLE_BYTES_DECOMPRESSED\",\"counterValue\":280},{\"counterName\":\"SHUFFLE_BYTES_TO_MEM\",\"counterValue\":192},{\"counterName\":\"SHUFFLE_BYTES_DISK_DIRECT\",\"counterValue\":128},{\"counterName\":\"SHUFFLE_PHASE_TIME\",\"counterValue\":7627},{\"counterName\":\"MERGE_PHASE_TIME\",\"counterValue\":7644},{\"counterName\":\"FIRST_EVENT_RECEIVED\",\"counterValue\":31},{\"counterName\":\"LAST_EVENT_RECEIVED\",\"counterValue\":7618}]}]},\"lastDataEvents\":{\"lastDataEvents\":[{\"TEZ_TASK_ATTEMPT_ID\":\"attempt_1471559811619_0002_1_00_000004_0\",\"ts\":1471570642938}]}}}\u0001\n",
      "{\"entity\":\"task_1471559811619_0002_1_01_000000\",\"entitytype\":\"TEZ_TASK_ID\",\"events\":[{\"ts\":1471570643474,\"eventtype\":\"TASK_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570634704,\"endTime\":1471570643474,\"timeTaken\":8770,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":382},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"HDFS_BYTES_WRITTEN\",\"counterValue\":215},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":4},{\"counterName\":\"HDFS_WRITE_OPS\",\"counterValue\":3}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"REDUCE_INPUT_GROUPS\",\"counterValue\":2},{\"counterName\":\"REDUCE_INPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":20},{\"counterName\":\"NUM_SHUFFLED_INPUTS\",\"counterValue\":10},{\"counterName\":\"MERGED_MAP_OUTPUTS\",\"counterValue\":10},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":179},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":1410},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":596115456},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4638515200},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":596115456},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_READ\",\"counterValue\":254},{\"counterName\":\"SHUFFLE_BYTES\",\"counterValue\":320},{\"counterName\":\"SHUFFLE_BYTES_DECOMPRESSED\",\"counterValue\":280},{\"counterName\":\"SHUFFLE_BYTES_TO_MEM\",\"counterValue\":192},{\"counterName\":\"SHUFFLE_BYTES_DISK_DIRECT\",\"counterValue\":128},{\"counterName\":\"SHUFFLE_PHASE_TIME\",\"counterValue\":7627},{\"counterName\":\"MERGE_PHASE_TIME\",\"counterValue\":7644},{\"counterName\":\"FIRST_EVENT_RECEIVED\",\"counterValue\":31},{\"counterName\":\"LAST_EVENT_RECEIVED\",\"counterValue\":7618}]}]},\"successfulAttemptId\":\"attempt_1471559811619_0002_1_01_000000_0\"}}\u0001\n",
      "{\"entity\":\"vertex_1471559811619_0002_1_01\",\"entitytype\":\"TEZ_VERTEX_ID\",\"events\":[{\"ts\":1471570643478,\"eventtype\":\"VERTEX_FINISHED\"}],\"otherinfo\":{\"endTime\":1471570643478,\"timeTaken\":17468,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":382},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"HDFS_BYTES_WRITTEN\",\"counterValue\":215},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":4},{\"counterName\":\"HDFS_WRITE_OPS\",\"counterValue\":3}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"REDUCE_INPUT_GROUPS\",\"counterValue\":2},{\"counterName\":\"REDUCE_INPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":20},{\"counterName\":\"NUM_SHUFFLED_INPUTS\",\"counterValue\":10},{\"counterName\":\"MERGED_MAP_OUTPUTS\",\"counterValue\":10},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":179},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":1410},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":596115456},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":4638515200},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":596115456},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_READ\",\"counterValue\":254},{\"counterName\":\"SHUFFLE_BYTES\",\"counterValue\":320},{\"counterName\":\"SHUFFLE_BYTES_DECOMPRESSED\",\"counterValue\":280},{\"counterName\":\"SHUFFLE_BYTES_TO_MEM\",\"counterValue\":192},{\"counterName\":\"SHUFFLE_BYTES_DISK_DIRECT\",\"counterValue\":128},{\"counterName\":\"SHUFFLE_PHASE_TIME\",\"counterValue\":7627},{\"counterName\":\"MERGE_PHASE_TIME\",\"counterValue\":7644},{\"counterName\":\"FIRST_EVENT_RECEIVED\",\"counterValue\":31},{\"counterName\":\"LAST_EVENT_RECEIVED\",\"counterValue\":7618}]}]},\"stats\":{\"firstTaskStartTime\":1471570634704,\"firstTasksToStart\":[\"task_1471559811619_0002_1_01_000000\"],\"lastTaskFinishTime\":1471570643474,\"lastTasksToFinish\":[\"task_1471559811619_0002_1_01_000000\"],\"minTaskDuration\":8770,\"maxTaskDuration\":8770,\"avgTaskDuration\":8770,\"shortestDurationTasks\":[\"task_1471559811619_0002_1_01_000000\"],\"longestDurationTasks\":[\"task_1471559811619_0002_1_01_000000\"]},\"numFailedTasks\":0,\"numSucceededTasks\":1,\"numKilledTaskAttempts\":0,\"numKilledTasks\":0,\"numFailedTaskAttempts\":0,\"numCompletedTasks\":1}}\u0001\n",
      "{\"entity\":\"dag_1471559811619_0002_1\",\"entitytype\":\"TEZ_DAG_ID\",\"events\":[{\"ts\":1471570643676,\"eventtype\":\"DAG_FINISHED\"}],\"otherinfo\":{\"startTime\":1471570625740,\"endTime\":1471570643676,\"timeTaken\":17936,\"status\":\"SUCCEEDED\",\"diagnostics\":\"\",\"counters\":{\"counterGroups\":[{\"counterGroupName\":\"org.apache.tez.common.counters.DAGCounter\",\"counters\":[{\"counterName\":\"NUM_SUCCEEDED_TASKS\",\"counterValue\":11},{\"counterName\":\"TOTAL_LAUNCHED_TASKS\",\"counterValue\":11},{\"counterName\":\"DATA_LOCAL_TASKS\",\"counterValue\":10},{\"counterName\":\"AM_CPU_MILLISECONDS\",\"counterValue\":2120},{\"counterName\":\"AM_GC_TIME_MILLIS\",\"counterValue\":30}]},{\"counterGroupName\":\"org.apache.tez.common.counters.FileSystemCounter\",\"counterGroupDisplayName\":\"File System Counters\",\"counters\":[{\"counterName\":\"FILE_BYTES_READ\",\"counterValue\":12302},{\"counterName\":\"FILE_BYTES_WRITTEN\",\"counterValue\":766},{\"counterName\":\"HDFS_BYTES_READ\",\"counterValue\":1180},{\"counterName\":\"HDFS_BYTES_WRITTEN\",\"counterValue\":215},{\"counterName\":\"HDFS_READ_OPS\",\"counterValue\":24},{\"counterName\":\"HDFS_WRITE_OPS\",\"counterValue\":3}]},{\"counterGroupName\":\"org.apache.tez.common.counters.TaskCounter\",\"counters\":[{\"counterName\":\"REDUCE_INPUT_GROUPS\",\"counterValue\":2},{\"counterName\":\"REDUCE_INPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"SPLIT_RAW_BYTES\",\"counterValue\":1460},{\"counterName\":\"SPILLED_RECORDS\",\"counterValue\":40},{\"counterName\":\"NUM_SHUFFLED_INPUTS\",\"counterValue\":10},{\"counterName\":\"MERGED_MAP_OUTPUTS\",\"counterValue\":10},{\"counterName\":\"GC_TIME_MILLIS\",\"counterValue\":826},{\"counterName\":\"CPU_MILLISECONDS\",\"counterValue\":22500},{\"counterName\":\"PHYSICAL_MEMORY_BYTES\",\"counterValue\":4624220160},{\"counterName\":\"VIRTUAL_MEMORY_BYTES\",\"counterValue\":50911064064},{\"counterName\":\"COMMITTED_HEAP_BYTES\",\"counterValue\":4624220160},{\"counterName\":\"INPUT_RECORDS_PROCESSED\",\"counterValue\":10},{\"counterName\":\"OUTPUT_RECORDS\",\"counterValue\":20},{\"counterName\":\"OUTPUT_BYTES\",\"counterValue\":180},{\"counterName\":\"OUTPUT_BYTES_WITH_OVERHEAD\",\"counterValue\":280},{\"counterName\":\"OUTPUT_BYTES_PHYSICAL\",\"counterValue\":320},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_WRITTEN\",\"counterValue\":126},{\"counterName\":\"ADDITIONAL_SPILLS_BYTES_READ\",\"counterValue\":254},{\"counterName\":\"SHUFFLE_CHUNK_COUNT\",\"counterValue\":10},{\"counterName\":\"SHUFFLE_BYTES\",\"counterValue\":320},{\"counterName\":\"SHUFFLE_BYTES_DECOMPRESSED\",\"counterValue\":280},{\"counterName\":\"SHUFFLE_BYTES_TO_MEM\",\"counterValue\":192},{\"counterName\":\"SHUFFLE_BYTES_DISK_DIRECT\",\"counterValue\":128},{\"counterName\":\"SHUFFLE_PHASE_TIME\",\"counterValue\":7627},{\"counterName\":\"MERGE_PHASE_TIME\",\"counterValue\":7644},{\"counterName\":\"FIRST_EVENT_RECEIVED\",\"counterValue\":31},{\"counterName\":\"LAST_EVENT_RECEIVED\",\"counterValue\":7618}]}]},\"completionApplicationAttemptId\":\"appattempt_1471559811619_0002_000001\",\"numFailedTasks\":0,\"numSucceededTasks\":11,\"numKilledTaskAttempts\":0,\"numKilledTasks\":0,\"numFailedTaskAttempts\":0,\"numCompletedTasks\":11}}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000004\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570644512,\"eventtype\":\"CONTAINER_STOPPED\"}],\"otherinfo\":{\"exitStatus\":0}}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000003\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570648763,\"eventtype\":\"CONTAINER_STOPPED\"}],\"otherinfo\":{\"exitStatus\":0}}\u0001\n",
      "{\"entity\":\"tez_container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"TEZ_CONTAINER_ID\",\"relatedEntities\":[{\"entity\":\"appattempt_1471559811619_0002_000001\",\"entitytype\":\"TEZ_APPLICATION_ATTEMPT\"},{\"entity\":\"container_e01_1471559811619_0002_01_000002\",\"entitytype\":\"containerId\"}],\"events\":[{\"ts\":1471570648763,\"eventtype\":\"CONTAINER_STOPPED\"}],\"otherinfo\":{\"exitStatus\":0}}\u0001\n",
      "End of LogType:history.txt.appattempt_1471559811619_0002_000001\n",
      "\n",
      "LogType:launch_container.sh\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:5564\n",
      "Log Contents:\n",
      "#!/bin/bash\n",
      "\n",
      "export NM_HTTP_PORT=\"8042\"\n",
      "export LOCAL_DIRS=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002\"\n",
      "export JAVA_HOME=\"/usr/java/jdk1.7.0_75\"\n",
      "export NM_AUX_SERVICE_mapreduce_shuffle=\"AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n",
      "\"\n",
      "export HADOOP_YARN_HOME=\"/usr/hdp/XXX.XXX.XXX.0-258/hadoop-yarn\"\n",
      "export CLASSPATH=\"$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:\"\n",
      "export HADOOP_TOKEN_FILE_LOCATION=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/container_tokens\"\n",
      "export NM_HOST=\"testvm004\"\n",
      "export APPLICATION_WEB_PROXY_BASE=\"/proxy/application_1471559811619_0002\"\n",
      "export JVM_PID=\"$$\"\n",
      "export USER=\"ansible\"\n",
      "export PWD=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001\"\n",
      "export CONTAINER_ID=\"container_e01_1471559811619_0002_01_000001\"\n",
      "export HOME=\"/home/\"\n",
      "export NM_PORT=\"45454\"\n",
      "export LOGNAME=\"ansible\"\n",
      "export TEZ_CLIENT_VERSION=\"XXX.XXX.XXX.2.4.2.0-258\"\n",
      "export APP_SUBMIT_TIME_ENV=\"1471570620028\"\n",
      "export MAX_APP_ATTEMPTS=\"2\"\n",
      "export HADOOP_CONF_DIR=\"/usr/hdp/current/hadoop-yarn-nodemanager/../hadoop/conf\"\n",
      "export MALLOC_ARENA_MAX=\"4\"\n",
      "export LD_LIBRARY_PATH=\"::/usr/hdp/XXX.XXX.XXX.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/XXX.XXX.XXX.0-258/hadoop/lib/native:$HADOOP_COMMON_HOME/lib/native/:$PWD\"\n",
      "export LOG_DIRS=\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001\"\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/tez-conf.pb\" \"tez-conf.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-dag.pb\" \"tez-dag.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\" \"tezlib\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\" \"job.split\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/16/job.xml\" \"job.xml\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/tez.session.local-resources.pb\" \"tez.session.local-resources.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/15/job.splitmetainfo\" \"job.splitmetainfo\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\" \"job.jar\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "# Creating copy of launch script\n",
      "cp \"launch_container.sh\" \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/launch_container.sh\"\n",
      "chmod 640 \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/launch_container.sh\"\n",
      "# Determining directory contents\n",
      "echo \"ls -l:\" 1>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "ls -l 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "echo \"find -L . -maxdepth 5 -ls:\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "find -L . -maxdepth 5 -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "echo \"broken symlinks(find -L . -maxdepth 5 -type l -ls):\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "find -L . -maxdepth 5 -type l -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/directory.info\"\n",
      "exec /bin/bash -c \"$JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx3840m  -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster 1>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/stdout 2>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/stderr \"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "End of LogType:launch_container.sh\n",
      "\n",
      "LogType:stderr\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:118\n",
      "Log Contents:\n",
      "2016-08-19 10:37:05 Running Dag: dag_1471559811619_0002_1\n",
      "2016-08-19 10:37:23 Completed Dag: dag_1471559811619_0002_1\n",
      "End of LogType:stderr\n",
      "\n",
      "LogType:stdout\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:118\n",
      "Log Contents:\n",
      "2016-08-19 10:37:05 Running Dag: dag_1471559811619_0002_1\n",
      "2016-08-19 10:37:23 Completed Dag: dag_1471559811619_0002_1\n",
      "End of LogType:stdout\n",
      "\n",
      "LogType:syslog\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:11362\n",
      "Log Contents:\n",
      "2016-08-19 10:37:01,669 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1471559811619_0002, attemptNum=1, AMContainerId=container_e01_1471559811619_0002_01_000001, jvmPid=18317, userFromEnv=ansible, cliSessionOption=false, pwd=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001, localDirs=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002, logDirs=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001\n",
      "2016-08-19 10:37:02,792 [INFO] [main] |app.DAGAppMaster|: Created DAGAppMaster for application appattempt_1471559811619_0002_000001, versionInfo=[ component=tez-dag, version=XXX.XXX.XXX.2.4.2.0-258, revision=fa554fdce4e3495e09a310e0a32bb34ccc5946ad, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160425-0638 ]\n",
      "2016-08-19 10:37:02,806 [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=XXX.XXX.XXX.2.4.2.0-258, AMVersion=XXX.XXX.XXX.2.4.2.0-258\n",
      "2016-08-19 10:37:03,829 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap\n",
      "2016-08-19 10:37:03,832 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker\n",
      "2016-08-19 10:37:03,835 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler\n",
      "2016-08-19 10:37:03,836 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher\n",
      "2016-08-19 10:37:03,836 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher\n",
      "2016-08-19 10:37:03,836 [INFO] [main] |app.DAGAppMaster|: Using concurrent dispatcher: false\n",
      "2016-08-19 10:37:03,838 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher\n",
      "2016-08-19 10:37:03,839 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher\n",
      "2016-08-19 10:37:03,840 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler\n",
      "2016-08-19 10:37:03,840 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler\n",
      "2016-08-19 10:37:03,866 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerEventHandler\n",
      "2016-08-19 10:37:03,867 [INFO] [main] |common.AsyncDispatcher|: Registering class org.apache.tez.dag.app.rm.NMCommunicatorEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherImpl\n",
      "2016-08-19 10:37:03,870 [INFO] [main] |node.AMNodeTracker|: blacklistDisablePercent is 33, blacklistingEnabled: true, maxTaskFailuresPerNode: 3\n",
      "2016-08-19 10:37:03,876 [ERROR] [main] |web.WebUIService|: Tez UI History URL is not set\n",
      "2016-08-19 10:37:03,878 [INFO] [main] |launcher.ContainerLauncherImpl|: Upper limit on the thread pool size is 500\n",
      "2016-08-19 10:37:03,878 [INFO] [main] |history.HistoryEventHandler|: Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService\n",
      "2016-08-19 10:37:03,884 [INFO] [main] |impl.SimpleHistoryLoggingService|: Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001\n",
      "2016-08-19 10:37:03,885 [INFO] [main] |impl.SimpleHistoryLoggingService|: Initializing SimpleHistoryLoggingService, logFileLocation=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/history.txt.appattempt_1471559811619_0002_000001, maxErrors=10\n",
      "2016-08-19 10:37:03,885 [INFO] [main] |recovery.RecoveryService|: RecoveryService initialized with recoveryPath=hdfs://hdfs-cluster/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/.tez/application_1471559811619_0002/recovery/1, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100\n",
      "2016-08-19 10:37:03,887 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:APP_LAUNCHED]: applicationId=application_1471559811619_0002, appSubmitTime=1471570620028, launchTime=1471570622790\n",
      "2016-08-19 10:37:03,888 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1471559811619_0002_000001, appSubmitTime=1471570620028, launchTime=1471570622790\n",
      "2016-08-19 10:37:03,900 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |impl.SimpleHistoryLoggingService|: Starting SimpleHistoryLoggingService\n",
      "2016-08-19 10:37:03,906 [INFO] [ServiceThread:DAGClientRPCServer] |ipc.CallQueueManager|: Using callQueue class java.util.concurrent.LinkedBlockingQueue\n",
      "2016-08-19 10:37:03,906 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskAttemptListenerImpTezDag] |ipc.CallQueueManager|: Using callQueue class java.util.concurrent.LinkedBlockingQueue\n",
      "2016-08-19 10:37:03,910 [INFO] [ServiceThread:org.apache.tez.dag.app.launcher.ContainerLauncherImpl] |impl.ContainerManagementProtocolProxy|: yarn.client.max-cached-nodemanagers-proxies : 0\n",
      "2016-08-19 10:37:03,931 [INFO] [Socket Reader #1 for port 36714] |ipc.Server|: Starting Socket Reader #1 for port 36714\n",
      "2016-08-19 10:37:03,933 [INFO] [Socket Reader #1 for port 46978] |ipc.Server|: Starting Socket Reader #1 for port 46978\n",
      "2016-08-19 10:37:03,976 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting\n",
      "2016-08-19 10:37:03,988 [INFO] [IPC Server Responder] |ipc.Server|: IPC Server Responder: starting\n",
      "2016-08-19 10:37:03,991 [INFO] [IPC Server listener on 46978] |ipc.Server|: IPC Server listener on 46978: starting\n",
      "2016-08-19 10:37:03,993 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at testvm004/XXX.XXX.XXX.73:46978\n",
      "2016-08-19 10:37:04,004 [INFO] [ServiceThread:org.apache.tez.dag.app.TaskAttemptListenerImpTezDag] |app.TaskAttemptListenerImpTezDag|: Instantiated TaskAttemptListener RPC at testvm004/XXX.XXX.XXX.73:36714\n",
      "2016-08-19 10:37:04,008 [INFO] [IPC Server listener on 36714] |ipc.Server|: IPC Server listener on 36714: starting\n",
      "2016-08-19 10:37:04,035 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog\n",
      "2016-08-19 10:37:04,050 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |server.AuthenticationFilter|: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.\n",
      "2016-08-19 10:37:04,057 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpRequestLog|: Http request log for http.requests. is not defined\n",
      "2016-08-19 10:37:04,063 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)\n",
      "2016-08-19 10:37:04,106 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context \n",
      "2016-08-19 10:37:04,106 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static\n",
      "2016-08-19 10:37:04,110 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /*\n",
      "2016-08-19 10:37:04,438 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules\n",
      "2016-08-19 10:37:04,443 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 46919\n",
      "2016-08-19 10:37:04,443 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: jetty-6.1.26.hwx\n",
      "2016-08-19 10:37:04,484 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Extract jar:file:/hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz/lib/hadoop-yarn-common-XXX.XXX.XXX.2.4.2.0-258.jar!/webapps/ to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/tmp/Jetty_0_0_0_0_46919_webapps____3upqpi/webapp\n",
      "2016-08-19 10:37:04,716 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@XXX.XXX.XXX.0:46919\n",
      "2016-08-19 10:37:04,716 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app  started at 46919\n",
      "2016-08-19 10:37:04,716 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at http://testvm004:46919/ui/\n",
      "2016-08-19 10:37:04,743 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerEventHandler] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 1000, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 5000, idleContainerMaxTimeout: 10000, sessionMinHeldContainers: 0\n",
      "2016-08-19 10:37:04,857 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1471559811619_0002_000001, startTime=1471570624855\n",
      "2016-08-19 10:37:04,857 [INFO] [main] |app.DAGAppMaster|: In Non-Session mode.\n",
      "2016-08-19 10:37:04,924 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: 12288 cpu: 1 taskAllocations: 0\n",
      "2016-08-19 10:37:04,924 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:12288, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0\n",
      "2016-08-19 10:37:04,928 [INFO] [Dispatcher thread {Central}] |node.AMNodeTracker|: Num cluster nodes = 4\n",
      "2016-08-19 10:37:04,939 [INFO] [main] |app.DAGAppMaster|: Merging AM credentials into DAG credentials\n",
      "2016-08-19 10:37:05,013 [INFO] [main] |app.DAGAppMaster|: Generating DAG graphviz file, dagId=dag_1471559811619_0002_1, filePath=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000001/dag_1471559811619_0002_1.dot\n",
      "2016-08-19 10:37:05,017 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1471559811619_0002_1\n",
      "End of LogType:syslog\n",
      "\n",
      "LogType:syslog_dag_1471559811619_0002_1\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:86862\n",
      "Log Contents:\n",
      "2016-08-19 10:37:05,017 [INFO] [main] |app.DAGAppMaster|: Running DAG: QuasiMonteCarlo\n",
      "2016-08-19 10:37:05,496 [INFO] [IPC Server handler 0 on 46978] |ipc.Server|: IPC Server handler 0 on 46978, call org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.getDAGStatus from XXX.XXX.XXX.72:48055 Call#129 Retry#0\n",
      "org.apache.tez.dag.api.TezException: No running dag at present\n",
      "\tat org.apache.tez.dag.api.client.DAGClientHandler.getDAG(DAGClientHandler.java:89)\n",
      "\tat org.apache.tez.dag.api.client.DAGClientHandler.getACLManager(DAGClientHandler.java:156)\n",
      "\tat org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:95)\n",
      "\tat org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7465)\n",
      "\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n",
      "\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)\n",
      "\tat java.security.AccessController.doPrivileged(Native Method)\n",
      "\tat javax.security.auth.Subject.doAs(Subject.java:415)\n",
      "\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)\n",
      "2016-08-19 10:37:05,515 [INFO] [IPC Server handler 0 on 46978] |ipc.Server|: IPC Server handler 0 on 46978, call org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB.getDAGStatus from XXX.XXX.XXX.72:48059 Call#132 Retry#0\n",
      "org.apache.tez.dag.api.TezException: No running dag at present\n",
      "\tat org.apache.tez.dag.api.client.DAGClientHandler.getDAG(DAGClientHandler.java:89)\n",
      "\tat org.apache.tez.dag.api.client.DAGClientHandler.getACLManager(DAGClientHandler.java:156)\n",
      "\tat org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:95)\n",
      "\tat org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7465)\n",
      "\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n",
      "\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)\n",
      "\tat java.security.AccessController.doPrivileged(Native Method)\n",
      "\tat javax.security.auth.Subject.doAs(Subject.java:415)\n",
      "\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)\n",
      "\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)\n",
      "2016-08-19 10:37:05,515 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:DAG_SUBMITTED]: dagID=dag_1471559811619_0002_1, submitTime=1471570624921\n",
      "2016-08-19 10:37:05,708 [INFO] [main] |impl.VertexImpl|: Default container context for vertex_1471559811619_0002_1_00 [initialmap]=LocalResources: [[ name=tezlib, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: -1 file: \"/apps/tez/tez.tar.gz\" } size: 58653910 timestamp: 1471531122434 type: ARCHIVE visibility: PUBLIC],[ name=job.jar, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.jar\" } size: 273458 timestamp: 1471570616641 type: FILE visibility: APPLICATION],[ name=job.split, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.split\" } size: 1467 timestamp: 1471570617108 type: FILE visibility: APPLICATION],[ name=tez-conf.pb, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: -1 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/.tez/application_1471559811619_0002/tez-conf.pb\" } size: 44350 timestamp: 1471570619579 type: FILE visibility: APPLICATION],[ name=job.xml, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.xml\" } size: 108179 timestamp: 1471570617762 type: FILE visibility: APPLICATION],[ name=job.splitmetainfo, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.splitmetainfo\" } size: 357 timestamp: 1471570617250 type: FILE visibility: APPLICATION],], environment: [[ SHELL=/bin/bash ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$PWD ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3840m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex: vertex_1471559811619_0002_1_00 [initialmap], Default Resources=<memory:4096, vCores:1>\n",
      "2016-08-19 10:37:05,708 [INFO] [main] |impl.VertexImpl|: Setting 1 additional inputs for vertexvertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:05,715 [INFO] [main] |impl.VertexImpl|: Default container context for vertex_1471559811619_0002_1_01 [finalreduce]=LocalResources: [[ name=tezlib, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: -1 file: \"/apps/tez/tez.tar.gz\" } size: 58653910 timestamp: 1471531122434 type: ARCHIVE visibility: PUBLIC],[ name=job.jar, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.jar\" } size: 273458 timestamp: 1471570616641 type: FILE visibility: APPLICATION],[ name=job.split, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.split\" } size: 1467 timestamp: 1471570617108 type: FILE visibility: APPLICATION],[ name=tez-conf.pb, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: -1 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/.tez/application_1471559811619_0002/tez-conf.pb\" } size: 44350 timestamp: 1471570619579 type: FILE visibility: APPLICATION],[ name=job.xml, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.xml\" } size: 108179 timestamp: 1471570617762 type: FILE visibility: APPLICATION],[ name=job.splitmetainfo, value=resource { scheme: \"hdfs\" host: \"hdfs-cluster\" port: 8020 file: \"/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/job.splitmetainfo\" } size: 357 timestamp: 1471570617250 type: FILE visibility: APPLICATION],], environment: [[ SHELL=/bin/bash ],[ CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[ HADOOP_ROOT_LOGGER=INFO,CLA ],[ LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$PWD ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3840m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex: vertex_1471559811619_0002_1_01 [finalreduce], Default Resources=<memory:4096, vCores:1>\n",
      "2016-08-19 10:37:05,716 [INFO] [main] |impl.VertexImpl|: Setting 1 additional outputs for vertex vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:05,734 [INFO] [main] |impl.DAGImpl|: Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder\n",
      "2016-08-19 10:37:05,737 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:DAG_INITIALIZED]: dagID=dag_1471559811619_0002_1, initTime=1471570625516\n",
      "2016-08-19 10:37:05,738 [INFO] [main] |impl.DAGImpl|: dag_1471559811619_0002_1 transitioned from NEW to INITED\n",
      "2016-08-19 10:37:05,741 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:DAG_STARTED]: dagID=dag_1471559811619_0002_1, startTime=1471570625740\n",
      "2016-08-19 10:37:05,742 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Added additional resources : [[]] to classpath\n",
      "2016-08-19 10:37:05,748 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1471559811619_0002_1 transitioned from INITED to RUNNING\n",
      "2016-08-19 10:37:05,748 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Root Inputs exist for Vertex: initialmap : {MRInput={InputName=MRInput}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=null}}\n",
      "2016-08-19 10:37:05,749 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting vertexManager to ImmediateStartVertexManager for vertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:05,762 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Creating 10 tasks for vertex: vertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:05,787 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Directly initializing vertex: vertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:05,792 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_INITIALIZED]: vertexName=initialmap, vertexId=vertex_1471559811619_0002_1_00, initRequestedTime=1471570625748, initedTime=1471570625789, numTasks=10, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, additionalInputsCount=1\n",
      "2016-08-19 10:37:05,792 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_00 [initialmap] transitioned from NEW to INITED due to event V_INIT\n",
      "2016-08-19 10:37:05,806 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_STARTED]: vertexId=vertex_1471559811619_0002_1_00, startRequestedTime=1471570625792, startedTime=1471570625792\n",
      "2016-08-19 10:37:05,810 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_00 [initialmap] transitioned from INITED to RUNNING due to event V_START\n",
      "2016-08-19 10:37:05,811 [INFO] [App Shared Pool - #0] |impl.ImmediateStartVertexManager|: Starting 10 in initialmap\n",
      "2016-08-19 10:37:05,813 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting user vertex manager plugin: org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager on vertex: vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:05,861 [INFO] [Dispatcher thread {Central}] |vertexmanager.ShuffleVertexManager|: Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1\n",
      "2016-08-19 10:37:05,861 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Creating 1 tasks for vertex: vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:05,862 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Directly initializing vertex: vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:05,862 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Setting up committers for vertex vertex_1471559811619_0002_1_01 [finalreduce], numAdditionalOutputs=1\n",
      "2016-08-19 10:37:05,862 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Instantiating committer for output=MROutput, vertex=vertex_1471559811619_0002_1_01 [finalreduce], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter\n",
      "2016-08-19 10:37:05,901 [INFO] [Dispatcher thread {Central}] |committer.MROutputCommitter|: Committer for finalreduce:MROutput using newmapred API\n",
      "2016-08-19 10:37:05,908 [INFO] [Dispatcher thread {Central}] |output.FileOutputCommitter|: File Output Committer Algorithm version is 1\n",
      "2016-08-19 10:37:05,908 [INFO] [Dispatcher thread {Central}] |output.FileOutputCommitter|: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
      "2016-08-19 10:37:05,911 [INFO] [Dispatcher thread {Central}] |committer.MROutputCommitter|: OutputCommitter for outputName=MROutput, vertexName=finalreduce, outputCommitterClass=org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter\n",
      "2016-08-19 10:37:05,974 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_INITIALIZED]: vertexName=finalreduce, vertexId=vertex_1471559811619_0002_1_01, initRequestedTime=1471570625812, initedTime=1471570625973, numTasks=1, processorName=org.apache.tez.mapreduce.processor.reduce.ReduceProcessor, additionalInputsCount=0\n",
      "2016-08-19 10:37:05,974 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_01 [finalreduce] transitioned from NEW to INITED due to event V_INIT\n",
      "2016-08-19 10:37:05,977 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source vertex started: vertex_1471559811619_0002_1_00 for vertex: vertex_1471559811619_0002_1_01 [finalreduce] numStartedSources: 1 numSources: 1\n",
      "2016-08-19 10:37:05,977 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Triggering start event for vertex: vertex_1471559811619_0002_1_01 [finalreduce] with distanceFromRoot: 1\n",
      "2016-08-19 10:37:05,999 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000000, scheduledTime=1471570625997, launchTime=1471570625997\n",
      "2016-08-19 10:37:05,999 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000001, scheduledTime=1471570625999, launchTime=1471570625999\n",
      "2016-08-19 10:37:06,003 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000002, scheduledTime=1471570626003, launchTime=1471570626003\n",
      "2016-08-19 10:37:06,004 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000003, scheduledTime=1471570626003, launchTime=1471570626003\n",
      "2016-08-19 10:37:06,005 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000004, scheduledTime=1471570626004, launchTime=1471570626004\n",
      "2016-08-19 10:37:06,007 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000005, scheduledTime=1471570626007, launchTime=1471570626007\n",
      "2016-08-19 10:37:06,008 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000006, scheduledTime=1471570626007, launchTime=1471570626007\n",
      "2016-08-19 10:37:06,009 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000007, scheduledTime=1471570626008, launchTime=1471570626008\n",
      "2016-08-19 10:37:06,009 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000008, scheduledTime=1471570626009, launchTime=1471570626009\n",
      "2016-08-19 10:37:06,010 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000009, scheduledTime=1471570626009, launchTime=1471570626009\n",
      "2016-08-19 10:37:06,016 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_STARTED]: vertexId=vertex_1471559811619_0002_1_01, startRequestedTime=1471570625977, startedTime=1471570626010\n",
      "2016-08-19 10:37:06,016 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_01 [finalreduce] transitioned from INITED to RUNNING due to event V_START\n",
      "2016-08-19 10:37:06,028 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,029 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,029 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,031 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,031 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,031 [INFO] [App Shared Pool - #0] |vertexmanager.ShuffleVertexManager|: OnVertexStarted vertex: finalreduce with 10 source tasks and 1 pending tasks\n",
      "2016-08-19 10:37:06,031 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,031 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,032 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,032 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,032 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,032 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,032 [INFO] [App Shared Pool - #1] |vertexmanager.ShuffleVertexManager|: Received configured notification : CONFIGURED for vertex: initialmap in vertex: finalreduce\n",
      "2016-08-19 10:37:06,032 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,035 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,036 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,037 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,043 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,043 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,043 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,050 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000000_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,050 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,050 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,050 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,051 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000001_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm006 rack: null\n",
      "2016-08-19 10:37:06,051 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,051 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,051 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000002_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000003_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm006 rack: null\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,052 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000004_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,053 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000005_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000006_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,054 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,055 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000007_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,055 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:06,055 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,055 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,055 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000008_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm004 rack: null\n",
      "2016-08-19 10:37:06,056 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:06,056 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:06,056 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:06,056 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_00_000009_0 with request: Capability[<memory:4096, vCores:1>]Priority[2] host: testvm006 rack: null\n",
      "2016-08-19 10:37:07,985 [INFO] [AMRM Heartbeater thread] |impl.AMRMClientImpl|: Received new token for : testvm007:45454\n",
      "2016-08-19 10:37:07,985 [INFO] [AMRM Heartbeater thread] |impl.AMRMClientImpl|: Received new token for : testvm005:45454\n",
      "2016-08-19 10:37:07,986 [INFO] [AMRM Heartbeater thread] |impl.AMRMClientImpl|: Received new token for : testvm006:45454\n",
      "2016-08-19 10:37:08,000 [INFO] [AMRM Callback Handler Thread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:08,001 [INFO] [AMRM Callback Handler Thread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:08,001 [INFO] [AMRM Callback Handler Thread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:08,005 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000002, task=attempt_1471559811619_0002_1_00_000000_0, containerHost=testvm007:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm007, honorLocalityFlags=true, reusedContainer=false, delayedContainers=2\n",
      "2016-08-19 10:37:08,005 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:08,005 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:08,006 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:08,006 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:08,013 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000003, task=attempt_1471559811619_0002_1_00_000001_0, containerHost=testvm005:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm005, honorLocalityFlags=true, reusedContainer=false, delayedContainers=1\n",
      "2016-08-19 10:37:08,013 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:08,013 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:08,013 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:08,013 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:08,014 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000004, task=attempt_1471559811619_0002_1_00_000002_0, containerHost=testvm006:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm006, honorLocalityFlags=true, reusedContainer=false, delayedContainers=0\n",
      "2016-08-19 10:37:08,014 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:08,014 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:08,014 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:08,014 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:08,038 [INFO] [TaskSchedulerAppCaller #0] |node.AMNodeTracker|: Adding new node: testvm007:45454\n",
      "2016-08-19 10:37:08,044 [INFO] [TaskSchedulerAppCaller #0] |node.AMNodeTracker|: Adding new node: testvm005:45454\n",
      "2016-08-19 10:37:08,044 [INFO] [TaskSchedulerAppCaller #0] |node.AMNodeTracker|: Adding new node: testvm006:45454\n",
      "2016-08-19 10:37:08,077 [INFO] [ContainerLauncher #0] |launcher.ContainerLauncherImpl|: Launching container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:08,077 [INFO] [ContainerLauncher #1] |launcher.ContainerLauncherImpl|: Launching container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:08,078 [INFO] [ContainerLauncher #2] |launcher.ContainerLauncherImpl|: Launching container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:08,078 [INFO] [ContainerLauncher #0] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm007:45454\n",
      "2016-08-19 10:37:08,108 [INFO] [ContainerLauncher #2] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm006:45454\n",
      "2016-08-19 10:37:08,110 [INFO] [ContainerLauncher #1] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm005:45454\n",
      "2016-08-19 10:37:08,178 [INFO] [ContainerLauncher #1] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_e01_1471559811619_0002_01_000003, launchTime=1471570628178\n",
      "2016-08-19 10:37:08,178 [INFO] [ContainerLauncher #0] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_e01_1471559811619_0002_01_000002, launchTime=1471570628178\n",
      "2016-08-19 10:37:08,181 [INFO] [ContainerLauncher #2] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_e01_1471559811619_0002_01_000004, launchTime=1471570628181\n",
      "2016-08-19 10:37:10,569 [INFO] [Socket Reader #1 for port 36714] |ipc.Server|: Auth successful for application_1471559811619_0002 (auth:SIMPLE)\n",
      "2016-08-19 10:37:10,596 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:10,597 [INFO] [IPC Server handler 0 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000002 given task: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:10,599 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000000_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000002] on NM: [testvm007:45454]\n",
      "2016-08-19 10:37:10,601 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0, startTime=1471570630596, containerId=container_e01_1471559811619_0002_01_000002, nodeId=testvm007:45454\n",
      "2016-08-19 10:37:10,601 [INFO] [Dispatcher thread {Central}] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120\n",
      "2016-08-19 10:37:11,079 [INFO] [Socket Reader #1 for port 36714] |ipc.Server|: Auth successful for application_1471559811619_0002 (auth:SIMPLE)\n",
      "2016-08-19 10:37:11,097 [INFO] [IPC Server handler 2 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000003 given task: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:11,097 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:11,097 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000001_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000003] on NM: [testvm005:45454]\n",
      "2016-08-19 10:37:11,098 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0, startTime=1471570631097, containerId=container_e01_1471559811619_0002_01_000003, nodeId=testvm005:45454\n",
      "2016-08-19 10:37:11,489 [INFO] [Socket Reader #1 for port 36714] |ipc.Server|: Auth successful for application_1471559811619_0002 (auth:SIMPLE)\n",
      "2016-08-19 10:37:11,509 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:11,509 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000002_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000004] on NM: [testvm006:45454]\n",
      "2016-08-19 10:37:11,510 [INFO] [IPC Server handler 1 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000004 given task: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:11,510 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0, startTime=1471570631509, containerId=container_e01_1471559811619_0002_01_000004, nodeId=testvm006:45454\n",
      "2016-08-19 10:37:12,614 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:12,624 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0, creationTime=1471570625996, allocationTime=1471570628065, startTime=1471570630596, finishTime=1471570632615, timeTaken=2019, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:12,626 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000002, task=attempt_1471559811619_0002_1_00_000003_0, containerHost=testvm007:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm007, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:12,626 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:12,626 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:12,626 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:12,627 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12,632 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000000, startTime=1471570630596, finishTime=1471570632628, timeTaken=2032, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000000_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1849, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=250, CPU_MILLISECONDS=2520, PHYSICAL_MEMORY_BYTES=290979840, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=290979840, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:12,633 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=1, completed=1, commits=0, err=null\n",
      "2016-08-19 10:37:12,633 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:12,642 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:12,642 [INFO] [IPC Server handler 16 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000002 given task: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:12,646 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000003_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000002] on NM: [testvm007:45454]\n",
      "2016-08-19 10:37:12,646 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0, startTime=1471570632642, containerId=container_e01_1471559811619_0002_01_000002, nodeId=testvm007:45454\n",
      "2016-08-19 10:37:13,746 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:13,747 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0, creationTime=1471570626003, allocationTime=1471570632633, startTime=1471570632642, finishTime=1471570633747, timeTaken=1105, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:13,748 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13,750 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000003, startTime=1471570632642, finishTime=1471570633748, timeTaken=1106, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000003_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1411, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=850, PHYSICAL_MEMORY_BYTES=396361728, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=396361728, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:13,750 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=2, completed=2, commits=0, err=null\n",
      "2016-08-19 10:37:13,750 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000003_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:13,753 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000002, task=attempt_1471559811619_0002_1_00_000004_0, containerHost=testvm007:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm007, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:13,753 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:13,753 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:13,754 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:13,850 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:13,851 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000004_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000002] on NM: [testvm007:45454]\n",
      "2016-08-19 10:37:13,851 [INFO] [IPC Server handler 23 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000002 given task: attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:13,851 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0, startTime=1471570633850, containerId=container_e01_1471559811619_0002_01_000002, nodeId=testvm007:45454\n",
      "2016-08-19 10:37:13,892 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:13,895 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0, creationTime=1471570625999, allocationTime=1471570628066, startTime=1471570631097, finishTime=1471570633892, timeTaken=2795, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:13,895 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13,896 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000003, task=attempt_1471559811619_0002_1_00_000005_0, containerHost=testvm005:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm005, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:13,896 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:13,896 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:13,896 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:13,899 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000001, startTime=1471570631097, finishTime=1471570633896, timeTaken=2799, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000001_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1703, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=189, CPU_MILLISECONDS=3680, PHYSICAL_MEMORY_BYTES=254803968, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=254803968, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:13,900 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=3, completed=3, commits=0, err=null\n",
      "2016-08-19 10:37:13,900 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000001_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:13,912 [INFO] [IPC Server handler 28 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000003 given task: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:13,914 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:13,914 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000005_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000003] on NM: [testvm005:45454]\n",
      "2016-08-19 10:37:13,915 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0, startTime=1471570633914, containerId=container_e01_1471559811619_0002_01_000003, nodeId=testvm005:45454\n",
      "2016-08-19 10:37:14,058 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:14,061 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0, creationTime=1471570626006, allocationTime=1471570633900, startTime=1471570633914, finishTime=1471570634059, timeTaken=145, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:14,061 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000003, task=attempt_1471559811619_0002_1_00_000006_0, containerHost=testvm005:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm005, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:14,061 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:14,061 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:14,061 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,062 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14,064 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000005, startTime=1471570633914, finishTime=1471570634062, timeTaken=148, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000005_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1119, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=170, PHYSICAL_MEMORY_BYTES=360185856, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=360185856, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:14,064 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=4, completed=4, commits=0, err=null\n",
      "2016-08-19 10:37:14,065 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000005_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:14,088 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:14,089 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0, creationTime=1471570626003, allocationTime=1471570628067, startTime=1471570631509, finishTime=1471570634089, timeTaken=2580, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:14,089 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000004, task=attempt_1471559811619_0002_1_00_000008_0, containerHost=testvm006:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm006, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:14,090 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:14,090 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:14,090 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,090 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14,094 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000002, startTime=1471570631509, finishTime=1471570634090, timeTaken=2581, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000002_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1557, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=208, CPU_MILLISECONDS=3400, PHYSICAL_MEMORY_BYTES=290455552, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=290455552, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:14,094 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=5, completed=5, commits=0, err=null\n",
      "2016-08-19 10:37:14,094 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000002_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:14,102 [INFO] [IPC Server handler 3 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000004 given task: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,102 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:14,103 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000008_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000004] on NM: [testvm006:45454]\n",
      "2016-08-19 10:37:14,103 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0, startTime=1471570634103, containerId=container_e01_1471559811619_0002_01_000004, nodeId=testvm006:45454\n",
      "2016-08-19 10:37:14,163 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,163 [INFO] [IPC Server handler 8 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000003 given task: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,163 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000006_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000003] on NM: [testvm005:45454]\n",
      "2016-08-19 10:37:14,164 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0, startTime=1471570634163, containerId=container_e01_1471559811619_0002_01_000003, nodeId=testvm005:45454\n",
      "2016-08-19 10:37:14,260 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:14,262 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0, creationTime=1471570626009, allocationTime=1471570634094, startTime=1471570634103, finishTime=1471570634260, timeTaken=157, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:14,263 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000004, task=attempt_1471559811619_0002_1_00_000009_0, containerHost=testvm006:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm006, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:14,263 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:14,263 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,263 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:14,264 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,277 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000008, startTime=1471570634103, finishTime=1471570634264, timeTaken=161, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000008_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=681, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=200, PHYSICAL_MEMORY_BYTES=395837440, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=395837440, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:14,278 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=6, completed=6, commits=0, err=null\n",
      "2016-08-19 10:37:14,278 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000008_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:14,317 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:14,319 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0, creationTime=1471570626007, allocationTime=1471570634064, startTime=1471570634163, finishTime=1471570634318, timeTaken=155, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:14,320 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,321 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000003, task=attempt_1471559811619_0002_1_00_000007_0, containerHost=testvm005:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm005, honorLocalityFlags=true, reusedContainer=true, delayedContainers=0\n",
      "2016-08-19 10:37:14,322 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm004 to /default-rack\n",
      "2016-08-19 10:37:14,322 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,322 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000006, startTime=1471570634163, finishTime=1471570634321, timeTaken=158, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000006_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=973, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=170, PHYSICAL_MEMORY_BYTES=465567744, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=465567744, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:14,322 [INFO] [TaskSchedulerEventHandlerThread] |util.RackResolver|: Resolved testvm007 to /default-rack\n",
      "2016-08-19 10:37:14,323 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=7, completed=7, commits=0, err=null\n",
      "2016-08-19 10:37:14,325 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000006_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:14,365 [INFO] [IPC Server handler 14 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000004 given task: attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:14,366 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm006 to /default-rack\n",
      "2016-08-19 10:37:14,367 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000009_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000004] on NM: [testvm006:45454]\n",
      "2016-08-19 10:37:14,368 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0, startTime=1471570634367, containerId=container_e01_1471559811619_0002_01_000004, nodeId=testvm006:45454\n",
      "2016-08-19 10:37:14,423 [INFO] [IPC Server handler 16 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000003 given task: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,425 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,425 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_00_000007_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000003] on NM: [testvm005:45454]\n",
      "2016-08-19 10:37:14,426 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0, startTime=1471570634425, containerId=container_e01_1471559811619_0002_01_000003, nodeId=testvm005:45454\n",
      "2016-08-19 10:37:14,598 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:14,601 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0, creationTime=1471570626008, allocationTime=1471570634327, startTime=1471570634425, finishTime=1471570634598, timeTaken=173, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:14,602 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,604 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000007, startTime=1471570634425, finishTime=1471570634602, timeTaken=177, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000007_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=827, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=260, PHYSICAL_MEMORY_BYTES=570949632, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=570949632, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:14,604 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=8, completed=8, commits=0, err=null\n",
      "2016-08-19 10:37:14,605 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000007_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:14,605 [INFO] [App Shared Pool - #0] |vertexmanager.ShuffleVertexManager|: Scheduling 1 tasks for vertex: finalreduce with totalTasks: 1. 8 source tasks completed out of 10. SourceTaskCompletedFraction: 0.8 min: 0.25 max: 0.75\n",
      "2016-08-19 10:37:14,606 [INFO] [App Shared Pool - #0] |impl.Edge|: Routing to begin for edge: EdgeInfo: sourceVertexName=initialmap, destinationVertexName=finalreduce. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedInputLegacy >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true\n",
      "2016-08-19 10:37:14,606 [INFO] [App Shared Pool - #0] |impl.VertexImpl|: Routing pending task events for vertex: vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:14,611 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_STARTED]: vertexName=finalreduce, taskId=task_1471559811619_0002_1_01_000000, scheduledTime=1471570634611, launchTime=1471570634611\n",
      "2016-08-19 10:37:14,612 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1471559811619_0002_1_01_000000_0 with request: Capability[<memory:4096, vCores:1>]Priority[5] host: null rack: null\n",
      "2016-08-19 10:37:14,616 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_e01_1471559811619_0002_01_000003, task=attempt_1471559811619_0002_1_01_000000_0, containerHost=testvm005:45454, containerPriority= 2, containerResources=<memory:4096, vCores:1>, localityMatchType=NodeLocal, matchedLocation=testvm005, honorLocalityFlags=true, reusedContainer=true, delayedContainers=1\n",
      "2016-08-19 10:37:14,704 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved testvm005 to /default-rack\n",
      "2016-08-19 10:37:14,705 [INFO] [Dispatcher thread {Central}] |impl.TaskAttemptImpl|: TaskAttempt: [attempt_1471559811619_0002_1_01_000000_0] started. Is using containerId: [container_e01_1471559811619_0002_01_000003] on NM: [testvm005:45454]\n",
      "2016-08-19 10:37:14,705 [INFO] [IPC Server handler 24 on 36714] |app.TaskAttemptListenerImpTezDag|: Container with id: container_e01_1471559811619_0002_01_000003 given task: attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:14,705 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_STARTED]: vertexName=finalreduce, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0, startTime=1471570634704, containerId=container_e01_1471559811619_0002_01_000003, nodeId=testvm005:45454\n",
      "2016-08-19 10:37:14,718 [INFO] [IPC Server handler 21 on 36714] |impl.VertexImpl|: Sending attempt_1471559811619_0002_1_01_000000_0 8 events [0,8) total 8 vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:18,002 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:18,003 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0, creationTime=1471570626009, allocationTime=1471570634278, startTime=1471570634367, finishTime=1471570638002, timeTaken=3635, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:18,003 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:18,004 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000009, startTime=1471570634367, finishTime=1471570638003, timeTaken=3636, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000009_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=535, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=3200, PHYSICAL_MEMORY_BYTES=501219328, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=501219328, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:18,004 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=9, completed=9, commits=0, err=null\n",
      "2016-08-19 10:37:18,004 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000009_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:18,076 [INFO] [IPC Server handler 1 on 36714] |impl.VertexImpl|: Sending attempt_1471559811619_0002_1_01_000000_0 1 events [8,9) total 9 vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:22,938 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1471559811619_0002_1_00, eventCount=1\n",
      "2016-08-19 10:37:22,940 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=initialmap, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0, creationTime=1471570626004, allocationTime=1471570633754, startTime=1471570633850, finishTime=1471570642939, timeTaken=9089, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:22,941 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:22,944 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=initialmap, taskId=task_1471559811619_0002_1_00_000004, startTime=1471570633850, finishTime=1471570642941, timeTaken=9091, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_00_000004_0, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1265, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=6640, PHYSICAL_MEMORY_BYTES=501743616, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=501743616, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1\n",
      "2016-08-19 10:37:22,944 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_00 [initialmap], tasks=10, failed=0, killed=0, success=10, completed=10, commits=0, err=null\n",
      "2016-08-19 10:37:22,948 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:22,974 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_FINISHED]: vertexName=initialmap, vertexId=vertex_1471559811619_0002_1_00, initRequestedTime=1471570625748, initedTime=1471570625789, startRequestedTime=1471570625792, startedTime=1471570625792, finishTime=1471570642948, timeTaken=17156, status=SUCCEEDED, diagnostics=, counters=Counters: 27, org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=10, File System Counters, FILE_BYTES_READ=11920, FILE_BYTES_WRITTEN=640, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=1180, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=20, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPLIT_RAW_BYTES=1460, SPILLED_RECORDS=20, GC_TIME_MILLIS=647, CPU_MILLISECONDS=21090, PHYSICAL_MEMORY_BYTES=4028104704, VIRTUAL_MEMORY_BYTES=46272548864, COMMITTED_HEAP_BYTES=4028104704, INPUT_RECORDS_PROCESSED=10, OUTPUT_RECORDS=20, OUTPUT_BYTES=180, OUTPUT_BYTES_WITH_OVERHEAD=280, OUTPUT_BYTES_PHYSICAL=320, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=10, vertexStats=firstTaskStartTime=1471570630596, firstTasksToStart=[ task_1471559811619_0002_1_00_000000 ], lastTaskFinishTime=1471570642939, lastTasksToFinish=[ task_1471559811619_0002_1_00_000004 ], minTaskDuration=145, maxTaskDuration=9089, avgTaskDuration=2185.3, numSuccessfulTasks=10, shortestDurationTasks=[ task_1471559811619_0002_1_00_000005 ], longestDurationTasks=[ task_1471559811619_0002_1_00_000004 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=10, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=10}\n",
      "2016-08-19 10:37:22,975 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_00 [initialmap] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED\n",
      "2016-08-19 10:37:22,975 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1471559811619_0002_1_01 [finalreduce] attempt: attempt_1471559811619_0002_1_00_000004_0 with state: SUCCEEDED vertexState: RUNNING\n",
      "2016-08-19 10:37:22,976 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1471559811619_0002_1_00 [initialmap] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2\n",
      "2016-08-19 10:37:22,976 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null\n",
      "2016-08-19 10:37:22,998 [INFO] [IPC Server handler 9 on 36714] |impl.VertexImpl|: Sending attempt_1471559811619_0002_1_01_000000_0 1 events [9,10) total 10 vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:23,327 [INFO] [IPC Server handler 21 on 36714] |app.TaskAttemptListenerImpTezDag|: Commit go/no-go request from attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23,327 [INFO] [IPC Server handler 21 on 36714] |impl.TaskImpl|: attempt_1471559811619_0002_1_01_000000_0 given a go for committing the task output.\n",
      "2016-08-19 10:37:23,474 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=finalreduce, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0, creationTime=1471570634611, allocationTime=1471570634617, startTime=1471570634704, finishTime=1471570643474, timeTaken=8770, status=SUCCEEDED, errorEnum=, diagnostics=\n",
      "2016-08-19 10:37:23,474 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Task succeeded with attempt attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23,478 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:TASK_FINISHED]: vertexName=finalreduce, taskId=task_1471559811619_0002_1_01_000000, startTime=1471570634704, finishTime=1471570643474, timeTaken=8770, status=SUCCEEDED, successfulAttemptID=attempt_1471559811619_0002_1_01_000000_0, diagnostics=, counters=Counters: 43, File System Counters, FILE_BYTES_READ=382, FILE_BYTES_WRITTEN=126, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=215, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=10, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=10, GC_TIME_MILLIS=179, CPU_MILLISECONDS=1410, PHYSICAL_MEMORY_BYTES=596115456, VIRTUAL_MEMORY_BYTES=4638515200, COMMITTED_HEAP_BYTES=596115456, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=126, ADDITIONAL_SPILLS_BYTES_READ=254, SHUFFLE_BYTES=320, SHUFFLE_BYTES_DECOMPRESSED=280, SHUFFLE_BYTES_TO_MEM=192, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=128, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7627, MERGE_PHASE_TIME=7644, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=7618, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0\n",
      "2016-08-19 10:37:23,478 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Task Completion: vertex_1471559811619_0002_1_01 [finalreduce], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null\n",
      "2016-08-19 10:37:23,478 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: All tasks have succeeded, vertex:vertex_1471559811619_0002_1_01 [finalreduce]\n",
      "2016-08-19 10:37:23,490 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:VERTEX_FINISHED]: vertexName=finalreduce, vertexId=vertex_1471559811619_0002_1_01, initRequestedTime=1471570625812, initedTime=1471570625973, startRequestedTime=1471570625977, startedTime=1471570626010, finishTime=1471570643478, timeTaken=17468, status=SUCCEEDED, diagnostics=, counters=Counters: 43, File System Counters, FILE_BYTES_READ=382, FILE_BYTES_WRITTEN=126, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=215, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=10, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=10, GC_TIME_MILLIS=179, CPU_MILLISECONDS=1410, PHYSICAL_MEMORY_BYTES=596115456, VIRTUAL_MEMORY_BYTES=4638515200, COMMITTED_HEAP_BYTES=596115456, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=126, ADDITIONAL_SPILLS_BYTES_READ=254, SHUFFLE_BYTES=320, SHUFFLE_BYTES_DECOMPRESSED=280, SHUFFLE_BYTES_TO_MEM=192, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=128, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7627, MERGE_PHASE_TIME=7644, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=7618, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, vertexStats=firstTaskStartTime=1471570634704, firstTasksToStart=[ task_1471559811619_0002_1_01_000000 ], lastTaskFinishTime=1471570643474, lastTasksToFinish=[ task_1471559811619_0002_1_01_000000 ], minTaskDuration=8770, maxTaskDuration=8770, avgTaskDuration=8770.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1471559811619_0002_1_01_000000 ], longestDurationTasks=[ task_1471559811619_0002_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}\n",
      "2016-08-19 10:37:23,490 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: vertex_1471559811619_0002_1_01 [finalreduce] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED\n",
      "2016-08-19 10:37:23,490 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Vertex vertex_1471559811619_0002_1_01 [finalreduce] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2\n",
      "2016-08-19 10:37:23,490 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null\n",
      "2016-08-19 10:37:23,490 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: No output committers for vertex: vertex_1471559811619_0002_1_00 [initialmap]\n",
      "2016-08-19 10:37:23,491 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: isCommitable of output:MROutput\n",
      "2016-08-19 10:37:23,492 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Start writing dag commit event, dag_1471559811619_0002_1\n",
      "2016-08-19 10:37:23,502 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1471559811619_0002_1, isCommitRepeatable=false\n",
      "2016-08-19 10:37:23,502 [INFO] [App Shared Pool - #0] |impl.DAGImpl|: Committing output: MROutput for vertex: vertex_1471559811619_0002_1_01 [finalreduce], outputName: MROutput\n",
      "2016-08-19 10:37:23,506 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1471559811619_0002_1 transitioned from RUNNING to COMMITTING\n",
      "2016-08-19 10:37:23,675 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Commit succeeded for output:outputName:MROutput of vertex/vertexGroup:finalreduce isVertexGroupOutput:false\n",
      "2016-08-19 10:37:23,675 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null\n",
      "2016-08-19 10:37:23,706 [INFO] [Dispatcher thread {Central}] |recovery.RecoveryService|: DAG completed, dagId=dag_1471559811619_0002_1, queueSize=0\n",
      "2016-08-19 10:37:23,755 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:DAG_FINISHED]: dagId=dag_1471559811619_0002_1, startTime=1471570625740, finishTime=1471570643676, timeTaken=17936, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=11, TOTAL_LAUNCHED_TASKS=11, DATA_LOCAL_TASKS=10, AM_CPU_MILLISECONDS=2120, AM_GC_TIME_MILLIS=30, File System Counters, FILE_BYTES_READ=12302, FILE_BYTES_WRITTEN=766, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=1180, HDFS_BYTES_WRITTEN=215, HDFS_READ_OPS=24, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=20, SPLIT_RAW_BYTES=1460, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=10, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=10, GC_TIME_MILLIS=826, CPU_MILLISECONDS=22500, PHYSICAL_MEMORY_BYTES=4624220160, VIRTUAL_MEMORY_BYTES=50911064064, COMMITTED_HEAP_BYTES=4624220160, INPUT_RECORDS_PROCESSED=10, OUTPUT_RECORDS=20, OUTPUT_BYTES=180, OUTPUT_BYTES_WITH_OVERHEAD=280, OUTPUT_BYTES_PHYSICAL=320, ADDITIONAL_SPILLS_BYTES_WRITTEN=126, ADDITIONAL_SPILLS_BYTES_READ=254, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=10, SHUFFLE_BYTES=320, SHUFFLE_BYTES_DECOMPRESSED=280, SHUFFLE_BYTES_TO_MEM=192, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=128, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7627, MERGE_PHASE_TIME=7644, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=7618, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0\n",
      "2016-08-19 10:37:23,756 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: DAG: dag_1471559811619_0002_1 finished with state: SUCCEEDED\n",
      "2016-08-19 10:37:23,757 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|: dag_1471559811619_0002_1 transitioned from COMMITTING to SUCCEEDED\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Not a session, AM will unregister as DAG has completed\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |rm.TaskSchedulerEventHandler|: TaskScheduler notified that it should unregister from RM\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1471559811619_0002_1_post\n",
      "End of LogType:syslog_dag_1471559811619_0002_1\n",
      "\n",
      "LogType:syslog_dag_1471559811619_0002_1_post\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:6524\n",
      "Log Contents:\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: On DAG completion. Old state: RUNNING new state: SUCCEEDED\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Shutting down on completion of dag:dag_1471559811619_0002_1\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked\n",
      "2016-08-19 10:37:23,758 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Handling DAGAppMaster shutdown\n",
      "2016-08-19 10:37:23,759 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down\n",
      "2016-08-19 10:37:24,509 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e01_1471559811619_0002_01_000004, containerExpiryTime=1471570644398, idleTimeout=5000, taskRequestsCount=0, heldContainers=3, delayedContainers=2, isNew=false\n",
      "2016-08-19 10:37:24,513 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:CONTAINER_STOPPED]: containerId=container_e01_1471559811619_0002_01_000004, stoppedTime=1471570644512, exitStatus=0\n",
      "2016-08-19 10:37:24,514 [INFO] [ContainerLauncher #3] |launcher.ContainerLauncherImpl|: Stopping container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:24,515 [INFO] [ContainerLauncher #3] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm006:45454\n",
      "2016-08-19 10:37:25,091 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_e01_1471559811619_0002_01_000004 exited with diagnostics set to Container failed, exitCode=-105. Container killed by the ApplicationMaster.\n",
      "Container killed on request. Exit code is 143\n",
      "Container exited with a non-zero exit code 143\n",
      "\n",
      "2016-08-19 10:37:28,761 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services\n",
      "2016-08-19 10:37:28,761 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler\n",
      "2016-08-19 10:37:28,761 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers\n",
      "2016-08-19 10:37:28,761 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests\n",
      "2016-08-19 10:37:28,762 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler\n",
      "2016-08-19 10:37:28,762 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService\n",
      "2016-08-19 10:37:28,762 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0\n",
      "2016-08-19 10:37:28,763 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning\n",
      "2016-08-19 10:37:28,763 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:CONTAINER_STOPPED]: containerId=container_e01_1471559811619_0002_01_000003, stoppedTime=1471570648763, exitStatus=0\n",
      "2016-08-19 10:37:28,763 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1471559811619_0002_1][Event:CONTAINER_STOPPED]: containerId=container_e01_1471559811619_0002_01_000002, stoppedTime=1471570648763, exitStatus=0\n",
      "2016-08-19 10:37:28,763 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream\n",
      "2016-08-19 10:37:28,768 [INFO] [ContainerLauncher #4] |launcher.ContainerLauncherImpl|: Stopping container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:28,768 [INFO] [ContainerLauncher #5] |launcher.ContainerLauncherImpl|: Stopping container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:28,768 [INFO] [ContainerLauncher #4] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm005:45454\n",
      "2016-08-19 10:37:28,770 [INFO] [ContainerLauncher #5] |impl.ContainerManagementProtocolProxy|: Opening proxy : testvm007:45454\n",
      "2016-08-19 10:37:28,805 [INFO] [AMShutdownThread] |impl.SimpleHistoryLoggingService|: Stopping SimpleHistoryLoggingService, eventQueueBacklog=0\n",
      "2016-08-19 10:37:28,805 [INFO] [HistoryEventHandlingThread] |impl.SimpleHistoryLoggingService|: EventQueue take interrupted. Returning\n",
      "2016-08-19 10:37:28,806 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted\n",
      "2016-08-19 10:37:28,808 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=, trackingURL=\n",
      "2016-08-19 10:37:28,816 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.\n",
      "2016-08-19 10:37:28,918 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM\n",
      "2016-08-19 10:37:28,919 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue\n",
      "java.lang.InterruptedException\n",
      "\tat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)\n",
      "\tat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)\n",
      "\tat java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\n",
      "\tat org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287)\n",
      "2016-08-19 10:37:28,925 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@XXX.XXX.XXX.0:0\n",
      "2016-08-19 10:37:28,925 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 36714\n",
      "2016-08-19 10:37:28,928 [INFO] [IPC Server listener on 36714] |ipc.Server|: Stopping IPC Server listener on 36714\n",
      "2016-08-19 10:37:28,928 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder\n",
      "2016-08-19 10:37:28,929 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 46978\n",
      "2016-08-19 10:37:28,929 [INFO] [IPC Server listener on 46978] |ipc.Server|: Stopping IPC Server listener on 46978\n",
      "2016-08-19 10:37:28,929 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder\n",
      "2016-08-19 10:37:28,959 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Completed deletion of tez scratch data dir, path=hdfs://hdfs-cluster/tmp/hadoop-yarn/staging/ansible/.staging/job_1471559811619_0002/.tez/application_1471559811619_0002\n",
      "2016-08-19 10:37:28,959 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye!\n",
      "2016-08-19 10:37:28,961 [INFO] [Thread-3] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked\n",
      "End of LogType:syslog_dag_1471559811619_0002_1_post\n",
      "\n",
      "\n",
      "\n",
      "Container: container_e01_1471559811619_0002_01_000003 on testvm005_45454\n",
      "==========================================================================\n",
      "LogType:directory.info\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:14414\n",
      "Log Contents:\n",
      "ls -l:\n",
      "total 48\n",
      "-rw-r--r-- 1 yarn yarn  143 Aug 19 10:37 container_tokens\n",
      "-rwx------ 1 yarn yarn  782 Aug 19 10:37 default_container_executor.sh\n",
      "-rwx------ 1 yarn yarn  728 Aug 19 10:37 default_container_executor_session.sh\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.jar -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\n",
      "lrwxrwxrwx 1 yarn yarn  117 Aug 19 10:37 job.split -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\n",
      "lrwxrwxrwx 1 yarn yarn  125 Aug 19 10:37 job.splitmetainfo -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.xml -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\n",
      "-rwx------ 1 yarn yarn 4953 Aug 19 10:37 launch_container.sh\n",
      "lrwxrwxrwx 1 yarn yarn  119 Aug 19 10:37 tez-conf.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\n",
      "lrwxrwxrwx 1 yarn yarn   60 Aug 19 10:37 tezlib -> /hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\n",
      "drwx--x--- 2 yarn yarn 4096 Aug 19 10:37 tmp\n",
      "find -L . -maxdepth 5 -ls:\n",
      "655685    4 drwx--x---   3 yarn     yarn         4096 Aug 19 10:37 .\n",
      "655688    4 -rw-r--r--   1 yarn     yarn           12 Aug 19 10:37 ./.container_tokens.crc\n",
      "655690    4 -rw-r--r--   1 yarn     yarn           48 Aug 19 10:37 ./.launch_container.sh.crc\n",
      "655678  108 -r-x------   1 yarn     yarn       108179 Aug 19 10:37 ./job.xml\n",
      "655687    4 -rw-r--r--   1 yarn     yarn          143 Aug 19 10:37 ./container_tokens\n",
      "655693    4 -rwx------   1 yarn     yarn          782 Aug 19 10:37 ./default_container_executor.sh\n",
      "655450    4 drwxr-xr-x   3 yarn     yarn         4096 Aug 18 23:45 ./tezlib\n",
      "655473  272 -r-xr-xr-x   1 yarn     yarn       275362 Apr 25 15:39 ./tezlib/tez-mapreduce-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655476  152 -r-xr-xr-x   1 yarn     yarn       154464 Apr 25 15:39 ./tezlib/tez-tests-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655472  532 -r-xr-xr-x   1 yarn     yarn       542434 Apr 25 15:39 ./tezlib/tez-runtime-library-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655474   48 -r-xr-xr-x   1 yarn     yarn        47672 Apr 25 15:39 ./tezlib/tez-examples-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655471  164 -r-xr-xr-x   1 yarn     yarn       164267 Apr 25 15:39 ./tezlib/tez-runtime-internals-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655481    8 -r-xr-xr-x   1 yarn     yarn         6334 Apr 25 15:41 ./tezlib/tez-yarn-timeline-cache-plugin-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655479   64 -r-xr-xr-x   1 yarn     yarn        62943 Apr 25 15:41 ./tezlib/tez-history-parser-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655470   68 -r-xr-xr-x   1 yarn     yarn        67195 Apr 25 15:38 ./tezlib/tez-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655482   24 -r-xr-xr-x   1 yarn     yarn        22125 Apr 25 15:42 ./tezlib/tez-yarn-timeline-history-with-fs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655478   28 -r-xr-xr-x   1 yarn     yarn        25622 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655483    4 drwxr-xr-x   2 yarn     yarn         4096 Aug 18 23:45 ./tezlib/lib\n",
      "655522  524 -r-xr-xr-x   1 yarn     yarn       533455 Apr 23 05:17 ./tezlib/lib/protobuf-java-2.5.0.jar\n",
      "655529   20 -r-xr-xr-x   1 yarn     yarn        16560 Apr 23 08:05 ./tezlib/lib/api-asn1-api-1.0.0-M20.jar\n",
      "655536 1444 -r-xr-xr-x   1 yarn     yarn      1475955 Apr 23 06:44 ./tezlib/lib/htrace-core-3.1.0-incubating.jar\n",
      "655519  976 -r-xr-xr-x   1 yarn     yarn       995968 Apr 23 05:17 ./tezlib/lib/snappy-java-XXX.XXX.XXX.1.jar\n",
      "655542  128 -r-xr-xr-x   1 yarn     yarn       130458 Apr 23 05:45 ./tezlib/lib/jersey-client-1.9.jar\n",
      "655527  676 -r-xr-xr-x   1 yarn     yarn       691479 Apr 23 08:05 ./tezlib/lib/apacheds-kerberos-codec-2.0.0-M15.jar\n",
      "655546    8 -r-xr-xr-x   1 yarn     yarn         4467 Apr 23 05:19 ./tezlib/lib/aopalliance-1.0.jar\n",
      "655510   64 -r-xr-xr-x   1 yarn     yarn        62050 Apr 23 05:34 ./tezlib/lib/commons-logging-1.1.3.jar\n",
      "655563  152 -r-xr-xr-x   1 yarn     yarn       153798 Apr 25 14:41 ./tezlib/lib/hadoop-azure-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655491   80 -r-xr-xr-x   1 yarn     yarn        81743 Apr 23 09:18 ./tezlib/lib/jettison-1.3.4.jar\n",
      "655551  312 -r-xr-xr-x   1 yarn     yarn       319099 Apr 23 05:19 ./tezlib/lib/okhttp-2.4.0.jar\n",
      "655559  848 -r-xr-xr-x   1 yarn     yarn       865838 Apr 23 05:19 ./tezlib/lib/jackson-databind-2.2.3.jar\n",
      "655560   36 -r-xr-xr-x   1 yarn     yarn        33483 Apr 23 05:19 ./tezlib/lib/jackson-annotations-2.2.3.jar\n",
      "655539  104 -r-xr-xr-x   1 yarn     yarn       105112 Apr 23 05:17 ./tezlib/lib/servlet-api-2.5.jar\n",
      "655514  144 -r-xr-xr-x   1 yarn     yarn       143602 Apr 23 05:17 ./tezlib/lib/commons-digester-1.8.jar\n",
      "655568   36 -r-xr-xr-x   1 yarn     yarn        34835 Apr 25 14:37 ./tezlib/lib/hadoop-yarn-server-web-proxy-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655525  424 -r-xr-xr-x   1 yarn     yarn       433368 Apr 23 07:01 ./tezlib/lib/httpclient-4.2.5.jar\n",
      "655570  100 -r-xr-xr-x   1 yarn     yarn        98771 Apr 25 14:58 ./tezlib/lib/hadoop-yarn-registry-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655561 11672 -r-xr-xr-x   1 yarn     yarn     11948376 Apr 23 08:19 ./tezlib/lib/aws-java-sdk-1.7.4.jar\n",
      "655566 1500 -r-xr-xr-x   1 yarn     yarn      1533812 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-core-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655484   12 -r-xr-xr-x   1 yarn     yarn         8869 Apr 23 06:23 ./tezlib/lib/slf4j-log4j12-1.7.5.jar\n",
      "655511  572 -r-xr-xr-x   1 yarn     yarn       582965 Apr 23 05:19 ./tezlib/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar\n",
      "655567  736 -r-xr-xr-x   1 yarn     yarn       751873 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655574  700 -r-xr-xr-x   1 yarn     yarn       714194 Apr 23 05:37 ./tezlib/lib/javassist-3.18.1-GA.jar\n",
      "655523  188 -r-xr-xr-x   1 yarn     yarn       190432 Apr 23 08:06 ./tezlib/lib/gson-2.2.4.jar\n",
      "655544  696 -r-xr-xr-x   1 yarn     yarn       710492 Apr 23 05:19 ./tezlib/lib/guice-3.0.jar\n",
      "655501  148 -r-xr-xr-x   1 yarn     yarn       147952 Apr 23 05:44 ./tezlib/lib/jersey-json-1.9.jar\n",
      "655564  652 -r-xr-xr-x   1 yarn     yarn       666763 Apr 23 05:19 ./tezlib/lib/azure-storage-2.2.0.jar\n",
      "655548  184 -r-xr-xr-x   1 yarn     yarn       185401 Apr 25 15:01 ./tezlib/lib/hadoop-yarn-client-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655557  532 -r-xr-xr-x   1 yarn     yarn       540751 Apr 23 05:19 ./tezlib/lib/jetty-6.1.26.hwx.jar\n",
      "655497   16 -r-xr-xr-x   1 yarn     yarn        15010 Apr 23 05:17 ./tezlib/lib/xmlenc-0.52.jar\n",
      "655573  336 -r-xr-xr-x   1 yarn     yarn       340884 Apr 23 05:19 ./tezlib/lib/fst-2.24.jar\n",
      "655538 1672 -r-xr-xr-x   1 yarn     yarn      1710975 Apr 25 14:57 ./tezlib/lib/hadoop-yarn-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655500  576 -r-xr-xr-x   1 yarn     yarn       588337 Apr 23 05:37 ./tezlib/lib/commons-collections-3.2.2.jar\n",
      "655521   96 -r-xr-xr-x   1 yarn     yarn        94672 Apr 23 05:17 ./tezlib/lib/xz-1.0.jar\n",
      "655535  268 -r-xr-xr-x   1 yarn     yarn       270342 Apr 23 06:44 ./tezlib/lib/curator-recipes-2.7.1.jar\n",
      "655549  736 -r-xr-xr-x   1 yarn     yarn       751238 Apr 23 09:18 ./tezlib/lib/commons-collections4-4.1.jar\n",
      "655496 1564 -r-xr-xr-x   1 yarn     yarn      1599627 Apr 23 06:43 ./tezlib/lib/commons-math3-3.1.1.jar\n",
      "655489  184 -r-xr-xr-x   1 yarn     yarn       185140 Apr 23 05:19 ./tezlib/lib/commons-io-2.4.jar\n",
      "655506  228 -r-xr-xr-x   1 yarn     yarn       232248 Apr 23 05:28 ./tezlib/lib/jackson-core-asl-1.9.13.jar\n",
      "655524   72 -r-xr-xr-x   1 yarn     yarn        71570 Apr 25 14:27 ./tezlib/lib/hadoop-auth-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655493 3420 -r-xr-xr-x   1 yarn     yarn      3498444 Apr 25 14:46 ./tezlib/lib/hadoop-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655485   28 -r-xr-xr-x   1 yarn     yarn        26084 Apr 23 05:34 ./tezlib/lib/slf4j-api-1.7.5.jar\n",
      "655528   44 -r-xr-xr-x   1 yarn     yarn        44925 Apr 23 08:05 ./tezlib/lib/apacheds-i18n-2.0.0-M15.jar\n",
      "655537 2060 -r-xr-xr-x   1 yarn     yarn      2108699 Apr 25 14:55 ./tezlib/lib/hadoop-yarn-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655515  188 -r-xr-xr-x   1 yarn     yarn       188671 Apr 23 05:14 ./tezlib/lib/commons-beanutils-1.7.0.jar\n",
      "655495   44 -r-xr-xr-x   1 yarn     yarn        41123 Apr 23 05:14 ./tezlib/lib/commons-cli-1.2.jar\n",
      "655504   24 -r-xr-xr-x   1 yarn     yarn        23346 Apr 23 05:14 ./tezlib/lib/stax-api-1.0-2.jar\n",
      "655550 8228 -r-xr-xr-x   1 yarn     yarn      8421957 Apr 25 14:50 ./tezlib/lib/hadoop-hdfs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655532 1172 -r-xr-xr-x   1 yarn     yarn      1199572 Apr 23 05:47 ./tezlib/lib/netty-3.6.2.Final.jar\n",
      "655503  104 -r-xr-xr-x   1 yarn     yarn       105134 Apr 23 05:17 ./tezlib/lib/jaxb-api-2.2.2.jar\n",
      "655533  184 -r-xr-xr-x   1 yarn     yarn       186273 Apr 23 06:43 ./tezlib/lib/curator-framework-2.7.1.jar\n",
      "655512  192 -r-xr-xr-x   1 yarn     yarn       192699 Apr 23 05:19 ./tezlib/lib/jackson-core-2.2.3.jar\n",
      "655526  224 -r-xr-xr-x   1 yarn     yarn       227275 Apr 23 07:14 ./tezlib/lib/httpcore-4.2.4.jar\n",
      "655556 1024 -r-xr-xr-x   1 yarn     yarn      1045744 Apr 23 05:19 ./tezlib/lib/leveldbjni-all-1.8.jar\n",
      "655488   36 -r-xr-xr-x   1 yarn     yarn        33031 Apr 23 09:18 ./tezlib/lib/jsr305-2.0.3.jar\n",
      "655513  292 -r-xr-xr-x   1 yarn     yarn       298829 Apr 23 05:17 ./tezlib/lib/commons-configuration-1.6.jar\n",
      "655558  104 -r-xr-xr-x   1 yarn     yarn       104578 Apr 25 14:41 ./tezlib/lib/hadoop-aws-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655562  616 -r-xr-xr-x   1 yarn     yarn       627814 Apr 23 08:19 ./tezlib/lib/joda-time-2.9.3.jar\n",
      "655499  268 -r-xr-xr-x   1 yarn     yarn       273370 Apr 23 05:17 ./tezlib/lib/commons-net-3.1.jar\n",
      "655520  236 -r-xr-xr-x   1 yarn     yarn       241367 Apr 23 06:43 ./tezlib/lib/commons-compress-1.4.1.jar\n",
      "655494   20 -r-xr-xr-x   1 yarn     yarn        17371 Apr 25 14:27 ./tezlib/lib/hadoop-annotations-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655534   68 -r-xr-xr-x   1 yarn     yarn        69500 Apr 23 06:43 ./tezlib/lib/curator-client-2.7.1.jar\n",
      "655545    4 -r-xr-xr-x   1 yarn     yarn         2497 Apr 23 05:19 ./tezlib/lib/javax.inject-1.jar\n",
      "655541  448 -r-xr-xr-x   1 yarn     yarn       458739 Apr 23 05:45 ./tezlib/lib/jersey-core-1.9.jar\n",
      "655487 1612 -r-xr-xr-x   1 yarn     yarn      1648200 Apr 23 05:18 ./tezlib/lib/guava-11.0.2.jar\n",
      "655492   28 -r-xr-xr-x   1 yarn     yarn        26514 Apr 23 05:19 ./tezlib/lib/stax-api-1.0.1.jar\n",
      "655565  404 -r-xr-xr-x   1 yarn     yarn       412739 Apr 23 06:23 ./tezlib/lib/commons-lang3-3.3.2.jar\n",
      "655531  776 -r-xr-xr-x   1 yarn     yarn       794058 Apr 25 14:27 ./tezlib/lib/zookeeper-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655517  300 -r-xr-xr-x   1 yarn     yarn       303139 Apr 25 14:28 ./tezlib/lib/avro-1.7.4.jar\n",
      "655543   64 -r-xr-xr-x   1 yarn     yarn        65012 Apr 23 05:19 ./tezlib/lib/guice-servlet-3.0.jar\n",
      "655552   64 -r-xr-xr-x   1 yarn     yarn        64661 Apr 23 05:19 ./tezlib/lib/okio-1.4.0.jar\n",
      "655555  192 -r-xr-xr-x   1 yarn     yarn       194354 Apr 23 05:14 ./tezlib/lib/xml-apis-1.3.04.jar\n",
      "655553 1740 -r-xr-xr-x   1 yarn     yarn      1779991 Apr 23 05:19 ./tezlib/lib/netty-all-4.0.23.Final.jar\n",
      "655498   60 -r-xr-xr-x   1 yarn     yarn        58160 Apr 23 08:05 ./tezlib/lib/commons-codec-1.4.jar\n",
      "655576   60 -r-xr-xr-x   1 yarn     yarn        60819 Apr 25 14:38 ./tezlib/lib/hadoop-yarn-server-timeline-plugins-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655575   44 -r-xr-xr-x   1 yarn     yarn        41755 Apr 23 05:51 ./tezlib/lib/objenesis-2.1.jar\n",
      "655572  244 -r-xr-xr-x   1 yarn     yarn       246782 Apr 25 15:00 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655508   20 -r-xr-xr-x   1 yarn     yarn        18336 Apr 23 08:06 ./tezlib/lib/jackson-jaxrs-1.9.13.jar\n",
      "655507  764 -r-xr-xr-x   1 yarn     yarn       780664 Apr 23 05:17 ./tezlib/lib/jackson-mapper-asl-1.9.13.jar\n",
      "655569  408 -r-xr-xr-x   1 yarn     yarn       413811 Apr 25 14:59 ./tezlib/lib/hadoop-yarn-server-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655540  176 -r-xr-xr-x   1 yarn     yarn       177487 Apr 23 05:19 ./tezlib/lib/jetty-util-6.1.26.hwx.jar\n",
      "655516  204 -r-xr-xr-x   1 yarn     yarn       206035 Apr 23 05:17 ./tezlib/lib/commons-beanutils-core-1.8.0.jar\n",
      "655547   16 -r-xr-xr-x   1 yarn     yarn        14786 Apr 23 05:45 ./tezlib/lib/jersey-guice-1.9.jar\n",
      "655502  872 -r-xr-xr-x   1 yarn     yarn       890168 Apr 23 05:17 ./tezlib/lib/jaxb-impl-2.2.3-1.jar\n",
      "655509   28 -r-xr-xr-x   1 yarn     yarn        27084 Apr 23 05:17 ./tezlib/lib/jackson-xc-1.9.13.jar\n",
      "655571  132 -r-xr-xr-x   1 yarn     yarn       134133 Apr 23 08:04 ./tezlib/lib/servlet-api-2.5-20081211.jar\n",
      "655490  280 -r-xr-xr-x   1 yarn     yarn       284220 Apr 23 05:14 ./tezlib/lib/commons-lang-2.6.jar\n",
      "655486  480 -r-xr-xr-x   1 yarn     yarn       489884 Apr 23 05:17 ./tezlib/lib/log4j-1.2.17.jar\n",
      "655518   32 -r-xr-xr-x   1 yarn     yarn        29555 Apr 23 05:17 ./tezlib/lib/paranamer-2.3.jar\n",
      "655554 1204 -r-xr-xr-x   1 yarn     yarn      1229125 Apr 23 05:14 ./tezlib/lib/xercesImpl-2.9.1.jar\n",
      "655505   64 -r-xr-xr-x   1 yarn     yarn        62983 Apr 23 05:17 ./tezlib/lib/activation-1.1.jar\n",
      "655530   80 -r-xr-xr-x   1 yarn     yarn        79912 Apr 23 08:05 ./tezlib/lib/api-util-1.0.0-M20.jar\n",
      "655480    8 -r-xr-xr-x   1 yarn     yarn         7784 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-with-acls-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655469  868 -r-xr-xr-x   1 yarn     yarn       888761 Apr 25 15:38 ./tezlib/tez-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655475 1196 -r-xr-xr-x   1 yarn     yarn      1222496 Apr 25 15:39 ./tezlib/tez-dag-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655477 1600 -r-xr-xr-x   1 yarn     yarn      1636310 Apr 25 15:40 ./tezlib/tez-ui-XXX.XXX.XXX.2.4.2.0-258.war\n",
      "655692    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor_session.sh.crc\n",
      "655669    4 -r-x------   1 yarn     yarn         1467 Aug 19 10:37 ./job.split\n",
      "655689    8 -rwx------   1 yarn     yarn         4953 Aug 19 10:37 ./launch_container.sh\n",
      "655691    4 -rwx------   1 yarn     yarn          728 Aug 19 10:37 ./default_container_executor_session.sh\n",
      "655675    4 -r-x------   1 yarn     yarn          357 Aug 19 10:37 ./job.splitmetainfo\n",
      "655672   44 -r-x------   1 yarn     yarn        44350 Aug 19 10:37 ./tez-conf.pb\n",
      "655694    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor.sh.crc\n",
      "655686    4 drwx--x---   2 yarn     yarn         4096 Aug 19 10:37 ./tmp\n",
      "655666  268 -r-x------   1 yarn     yarn       273458 Aug 19 10:37 ./job.jar\n",
      "broken symlinks(find -L . -maxdepth 5 -type l -ls):\n",
      "End of LogType:directory.info\n",
      "\n",
      "LogType:launch_container.sh\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:4953\n",
      "Log Contents:\n",
      "#!/bin/bash\n",
      "\n",
      "export SHELL=\"/bin/bash\"\n",
      "export NM_HTTP_PORT=\"8042\"\n",
      "export LOCAL_DIRS=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002\"\n",
      "export JAVA_HOME=\"/usr/java/jdk1.7.0_75\"\n",
      "export NM_AUX_SERVICE_mapreduce_shuffle=\"AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n",
      "\"\n",
      "export HADOOP_YARN_HOME=\"/usr/hdp/XXX.XXX.XXX.0-258/hadoop-yarn\"\n",
      "export CLASSPATH=\"$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:\"\n",
      "export HADOOP_TOKEN_FILE_LOCATION=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/container_tokens\"\n",
      "export NM_HOST=\"testvm005\"\n",
      "export HADOOP_ROOT_LOGGER=\"INFO,CLA\"\n",
      "export JVM_PID=\"$$\"\n",
      "export USER=\"ansible\"\n",
      "export PWD=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003\"\n",
      "export CONTAINER_ID=\"container_e01_1471559811619_0002_01_000003\"\n",
      "export HOME=\"/home/\"\n",
      "export NM_PORT=\"45454\"\n",
      "export LOGNAME=\"ansible\"\n",
      "export HADOOP_CONF_DIR=\"/usr/hdp/current/hadoop-yarn-nodemanager/../hadoop/conf\"\n",
      "export MALLOC_ARENA_MAX=\"4\"\n",
      "export LD_LIBRARY_PATH=\"$PWD:$HADOOP_COMMON_HOME/lib/native:$PWD\"\n",
      "export LOG_DIRS=\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003\"\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\" \"tezlib\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\" \"job.split\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\" \"job.xml\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\" \"tez-conf.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\" \"job.splitmetainfo\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\" \"job.jar\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "# Creating copy of launch script\n",
      "cp \"launch_container.sh\" \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/launch_container.sh\"\n",
      "chmod 640 \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/launch_container.sh\"\n",
      "# Determining directory contents\n",
      "echo \"ls -l:\" 1>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "ls -l 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "echo \"find -L . -maxdepth 5 -ls:\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "find -L . -maxdepth 5 -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "echo \"broken symlinks(find -L . -maxdepth 5 -type l -ls):\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "find -L . -maxdepth 5 -type l -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/directory.info\"\n",
      "exec /bin/bash -c \"$JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3840m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003 -Dtez.root.logger=INFO,CLA  -Djava.io.tmpdir=$PWD/tmp org.apache.tez.runtime.task.TezChild XXX.XXX.XXX.73 36714 container_e01_1471559811619_0002_01_000003 application_1471559811619_0002 1 1>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/stdout 2>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/stderr \"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "End of LogType:launch_container.sh\n",
      "\n",
      "LogType:stderr\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:940\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13 Completed running task attempt: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23 Completed running task attempt: attempt_1471559811619_0002_1_01_000000_0\n",
      "End of LogType:stderr\n",
      "\n",
      "LogType:stdout\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:940\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13 Completed running task attempt: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23 Completed running task attempt: attempt_1471559811619_0002_1_01_000000_0\n",
      "End of LogType:stdout\n",
      "\n",
      "LogType:syslog\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:3281\n",
      "Log Contents:\n",
      "2016-08-19 10:37:08,871 [INFO] [main] |task.TezChild|: TezChild starting with PID=18156, containerIdentifier=container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:10,265 [WARN] [main] |util.NativeCodeLoader|: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "2016-08-19 10:37:10,883 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:11,221 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:11,225 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13,905 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:13,917 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:13,918 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14,162 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:14,165 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,166 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,422 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:14,427 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,428 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,702 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "2016-08-19 10:37:14,709 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:14,710 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23,578 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000003\n",
      "End of LogType:syslog\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000001_0\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:8586\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11,248 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=1\n",
      "2016-08-19 10:37:11,345 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:11,345 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000001_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:11,357 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:11,401 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:11,609 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000001_0\n",
      "2016-08-19 10:37:11,639 [INFO] [I/O Setup 0 Initialize: {MRInput}] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120\n",
      "2016-08-19 10:37:11,649 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:11,649 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:11,986 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:11,986 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:11,986 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:11,987 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:11,994 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:12,137 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:12,141 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:12,141 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:12,142 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:12,142 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:12,143 [INFO] [I/O Setup 1 Start: {MRInput}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: MRInput\n",
      "2016-08-19 10:37:12,143 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: MRInput being auto started by the framework. Subsequent instances will not be auto-started\n",
      "2016-08-19 10:37:12,144 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1\n",
      "2016-08-19 10:37:12,144 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start\n",
      "2016-08-19 10:37:12,144 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:12,146 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:12,146 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000001_0_10001\n",
      "2016-08-19 10:37:12,193 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@3ebfe943, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:12,199 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:12,570 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:12,750 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:12,770 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/job.split\n",
      "2016-08-19 10:37:13,838 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:13,839 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part1:0+118\n",
      "2016-08-19 10:37:13,843 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000001_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:13,843 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0\n",
      "2016-08-19 10:37:13,848 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:13,848 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:13,848 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:13,853 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:13,853 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000001_0_10003_0/file.out\n",
      "2016-08-19 10:37:13,873 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:13,878 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000001_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:13,894 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000001_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1703, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=189, CPU_MILLISECONDS=3680, PHYSICAL_MEMORY_BYTES=254803968, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=254803968, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:13,904 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000001_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000005_0\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:7806\n",
      "Log Contents:\n",
      "2016-08-19 10:37:13,919 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:13,919 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000005_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:13,920 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:13,920 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:13,941 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000005_0\n",
      "2016-08-19 10:37:13,942 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:13,942 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:13,952 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:13,952 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:13,952 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:13,967 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:13,967 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:13,967 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:13,967 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:13,968 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:13,968 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:13,968 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:13,969 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:13,969 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:13,982 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:13,982 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000005_0_10004\n",
      "2016-08-19 10:37:13,983 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@2ae5b5a3, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:13,983 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:14,011 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:14,013 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:14,015 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/job.split\n",
      "2016-08-19 10:37:14,045 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:14,045 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part5:0+118\n",
      "2016-08-19 10:37:14,046 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000005_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:14,046 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0\n",
      "2016-08-19 10:37:14,046 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:14,046 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:14,046 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:14,047 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:14,047 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000005_0_10006_0/file.out\n",
      "2016-08-19 10:37:14,052 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:14,054 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000005_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:14,059 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000005_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1119, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=170, PHYSICAL_MEMORY_BYTES=360185856, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=360185856, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:14,161 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000005_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000006_0\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:7805\n",
      "Log Contents:\n",
      "2016-08-19 10:37:14,166 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:14,166 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000006_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:14,166 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:14,167 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,184 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000006_0\n",
      "2016-08-19 10:37:14,184 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:14,184 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:14,195 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:14,195 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:14,195 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:14,202 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:14,203 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:14,203 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:14,203 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:14,203 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:14,204 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:14,204 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:14,204 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:14,204 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:14,208 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,208 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000006_0_10007\n",
      "2016-08-19 10:37:14,210 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@5c75e98d, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:14,211 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:14,262 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:14,263 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:14,266 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/job.split\n",
      "2016-08-19 10:37:14,296 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:14,296 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part6:0+118\n",
      "2016-08-19 10:37:14,307 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000006_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:14,307 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0\n",
      "2016-08-19 10:37:14,307 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:14,307 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:14,307 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:14,309 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:14,309 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000006_0_10009_0/file.out\n",
      "2016-08-19 10:37:14,312 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:14,313 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000006_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:14,319 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000006_0: Counters: 26 [[File System Counters FILE_BYTES_READ=973, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=170, PHYSICAL_MEMORY_BYTES=465567744, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=465567744, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:14,421 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000006_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000007_0\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:7805\n",
      "Log Contents:\n",
      "2016-08-19 10:37:14,428 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:14,429 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000007_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:14,429 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:14,430 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,446 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000007_0\n",
      "2016-08-19 10:37:14,446 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:14,446 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:14,455 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:14,463 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:14,463 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:14,464 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:14,471 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,472 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000007_0_10010\n",
      "2016-08-19 10:37:14,474 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@4371a1a4, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:14,474 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:14,527 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:14,534 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:14,537 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000003/job.split\n",
      "2016-08-19 10:37:14,572 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:14,572 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part7:0+118\n",
      "2016-08-19 10:37:14,573 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000007_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:14,574 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0\n",
      "2016-08-19 10:37:14,574 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:14,574 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:14,574 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:14,576 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:14,576 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000007_0_10012_0/file.out\n",
      "2016-08-19 10:37:14,585 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:14,587 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000007_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:14,600 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000007_0: Counters: 26 [[File System Counters FILE_BYTES_READ=827, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=260, PHYSICAL_MEMORY_BYTES=570949632, VIRTUAL_MEMORY_BYTES=4626919424, COMMITTED_HEAP_BYTES=570949632, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:14,701 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000007_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_01_000000_0\n",
      "Log Upload Time:Fri Aug 19 10:37:30 +0900 2016\n",
      "LogLength:22191\n",
      "Log Contents:\n",
      "2016-08-19 10:37:14,712 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:14,713 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: finalreduce, VertexParallelism: 1, TaskAttemptID:attempt_1471559811619_0002_1_01_000000_0, processorName=org.apache.tez.mapreduce.processor.reduce.ReduceProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=initialmap, physicalEdgeCount=10, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=MROutput, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutputLegacy }}, ]\n",
      "2016-08-19 10:37:14,713 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:14,715 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:14,730 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_r_000000_0\n",
      "2016-08-19 10:37:14,740 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:14,740 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:14,747 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1471559811619_0002_1_01_000000_0, eventCount=8 fromEventId=0 nextFromEventId=8\n",
      "2016-08-19 10:37:14,753 [INFO] [I/O Setup 0 Initialize: {initialmap}] |orderedgrouped.MergeManager|: Initial Memory required for SHUFFLE_BUFFER=2505519872 based on INPUT_BUFFER_FACTOR=0.7,  for final merged output=0, using factor: 0.0\n",
      "2016-08-19 10:37:14,763 [INFO] [I/O Setup 1 Initialize: {MROutput}] |output.FileOutputCommitter|: File Output Committer Algorithm version is 1\n",
      "2016-08-19 10:37:14,763 [INFO] [I/O Setup 1 Initialize: {MROutput}] |output.FileOutputCommitter|: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
      "2016-08-19 10:37:14,768 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:14,947 [INFO] [I/O Setup 1 Initialize: {MROutput}] |output.MROutput|: MROutput: outputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat, using newmapreduce API=true\n",
      "2016-08-19 10:37:14,947 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:14,947 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MROutput:OUTPUT:0:org.apache.tez.mapreduce.output.MROutputLegacy], [initialmap:INPUT:2505519872:org.apache.tez.runtime.library.input.OrderedGroupedInputLegacy]\n",
      "2016-08-19 10:37:14,948 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:14,948 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:14,948 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 2505519872, TotalRequestedScaled: 2.3127875741538463E9, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.70\n",
      "2016-08-19 10:37:14,949 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MROutput:org.apache.tez.mapreduce.output.MROutputLegacy:OUTPUT:0:0], [initialmap:org.apache.tez.runtime.library.input.OrderedGroupedInputLegacy:INPUT:2505519872:2398140497]\n",
      "2016-08-19 10:37:14,949 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:14,949 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: initialmap being auto started by the framework. Subsequent instances will not be auto-started\n",
      "2016-08-19 10:37:14,949 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1\n",
      "2016-08-19 10:37:14,949 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start\n",
      "2016-08-19 10:37:15,383 [INFO] [I/O Setup 0 Start: {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shuffle assigned with 10 inputs, codec: NoneifileReadAhead: true\n",
      "2016-08-19 10:37:15,389 [INFO] [I/O Setup 0 Start: {initialmap}] |orderedgrouped.ShuffleScheduler|: ShuffleScheduler running for sourceVertex: initialmap with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=5, abortFailureLimit=15, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true, maxTaskOutputAtOnce=20\n",
      "2016-08-19 10:37:15,393 [INFO] [I/O Setup 0 Start: {initialmap}] |orderedgrouped.MergeManager|: initialmap: MergerManager: memoryLimit=2398140497, maxSingleShuffleLimit=599535104, mergeThreshold=1582772736, ioSortFactor=10, postMergeMem=0, memToMemMergeOutputsThreshold=10\n",
      "2016-08-19 10:37:15,399 [INFO] [I/O Setup 0 Start: {initialmap}] |orderedgrouped.Shuffle|: initialmap: Num fetchers being started: 5\n",
      "2016-08-19 10:37:15,406 [INFO] [I/O Setup 0 Start: {initialmap}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: initialmap\n",
      "2016-08-19 10:37:15,406 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:15,408 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:15,408 [INFO] [TezChild] |reduce.ReduceProcessor|: Running reduce: attempt_1471559811619_0002_1_01_000000_0_10013\n",
      "2016-08-19 10:37:15,414 [INFO] [TezTaskEventRouter{attempt_1471559811619_0002_1_01_000000_0}] |orderedgrouped.ShuffleInputEventHandlerOrderedGrouped|: initialmap: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0\n",
      "2016-08-19 10:37:15,429 [INFO] [fetcher {initialmap} #2] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {1, 0, attempt_1471559811619_0002_1_00_000001_0_10003} to DISK_DIRECT, csize=32, dsize=28, EndTime=1471570635429, TimeTaken=4, Rate=0.01 MB/s\n",
      "2016-08-19 10:37:15,430 [INFO] [fetcher {initialmap} #2] |orderedgrouped.ShuffleScheduler|: copy(1 (spillsFetched=1) of 10. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)\n",
      "2016-08-19 10:37:15,431 [INFO] [fetcher {initialmap} #2] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {5, 0, attempt_1471559811619_0002_1_00_000005_0_10006} to DISK_DIRECT, csize=32, dsize=28, EndTime=1471570635431, TimeTaken=1, Rate=0.03 MB/s\n",
      "2016-08-19 10:37:15,431 [INFO] [fetcher {initialmap} #2] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {6, 0, attempt_1471559811619_0002_1_00_000006_0_10009} to DISK_DIRECT, csize=32, dsize=28, EndTime=1471570635431, TimeTaken=0, Rate=0.00 MB/s\n",
      "2016-08-19 10:37:15,432 [INFO] [fetcher {initialmap} #2] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {7, 0, attempt_1471559811619_0002_1_00_000007_0_10012} to DISK_DIRECT, csize=32, dsize=28, EndTime=1471570635432, TimeTaken=1, Rate=0.03 MB/s\n",
      "2016-08-19 10:37:15,493 [INFO] [fetcher {initialmap} #4] |shuffle.HttpConnection|: for url=http://testvm007:13562/mapOutput?job=job_1471559811619_0002&reduce=0&map=attempt_1471559811619_0002_1_00_000000_0_10003,attempt_1471559811619_0002_1_00_000003_0_10006 sent hash and receievd reply 0 ms\n",
      "2016-08-19 10:37:15,641 [INFO] [fetcher {initialmap} #4] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->28, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000000_0_10003, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:15,641 [INFO] [fetcher {initialmap} #4] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {0, 0, attempt_1471559811619_0002_1_00_000000_0_10003} to MEMORY, csize=32, dsize=28, EndTime=1471570635641, TimeTaken=147, Rate=0.00 MB/s\n",
      "2016-08-19 10:37:15,641 [INFO] [fetcher {initialmap} #4] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 2, commitMemory -> 28, usedMemory ->56, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=3], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000003_0_10006, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:15,641 [INFO] [fetcher {initialmap} #4] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {3, 0, attempt_1471559811619_0002_1_00_000003_0_10006} to MEMORY, csize=32, dsize=28, EndTime=1471570635641, TimeTaken=0, Rate=0.00 MB/s\n",
      "2016-08-19 10:37:15,734 [INFO] [fetcher {initialmap} #1] |shuffle.HttpConnection|: for url=http://testvm006:13562/mapOutput?job=job_1471559811619_0002&reduce=0&map=attempt_1471559811619_0002_1_00_000002_0_10003,attempt_1471559811619_0002_1_00_000008_0_10006 sent hash and receievd reply 0 ms\n",
      "2016-08-19 10:37:16,005 [INFO] [fetcher {initialmap} #1] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 3, commitMemory -> 56, usedMemory ->84, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=2], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000002_0_10003, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:16,005 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {2, 0, attempt_1471559811619_0002_1_00_000002_0_10003} to MEMORY, csize=32, dsize=28, EndTime=1471570636005, TimeTaken=271, Rate=0.00 MB/s\n",
      "2016-08-19 10:37:16,248 [INFO] [fetcher {initialmap} #1] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 4, commitMemory -> 84, usedMemory ->112, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=8], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000008_0_10006, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:16,248 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {8, 0, attempt_1471559811619_0002_1_00_000008_0_10006} to MEMORY, csize=32, dsize=28, EndTime=1471570636248, TimeTaken=242, Rate=0.00 MB/s\n",
      "2016-08-19 10:37:18,078 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1471559811619_0002_1_01_000000_0, eventCount=1 fromEventId=8 nextFromEventId=9\n",
      "2016-08-19 10:37:18,084 [INFO] [fetcher {initialmap} #1] |shuffle.HttpConnection|: for url=http://testvm006:13562/mapOutput?job=job_1471559811619_0002&reduce=0&map=attempt_1471559811619_0002_1_00_000009_0_10009 sent hash and receievd reply 0 ms\n",
      "2016-08-19 10:37:18,085 [INFO] [fetcher {initialmap} #1] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 5, commitMemory -> 112, usedMemory ->140, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=9], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000009_0_10009, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:18,085 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {9, 0, attempt_1471559811619_0002_1_00_000009_0_10009} to MEMORY, csize=32, dsize=28, EndTime=1471570638085, TimeTaken=1, Rate=0.03 MB/s\n",
      "2016-08-19 10:37:23,001 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1471559811619_0002_1_01_000000_0, eventCount=1 fromEventId=9 nextFromEventId=10\n",
      "2016-08-19 10:37:23,008 [INFO] [fetcher {initialmap} #1] |shuffle.HttpConnection|: for url=http://testvm007:13562/mapOutput?job=job_1471559811619_0002&reduce=0&map=attempt_1471559811619_0002_1_00_000004_0_10009 sent hash and receievd reply 0 ms\n",
      "2016-08-19 10:37:23,009 [INFO] [fetcher {initialmap} #1] |orderedgrouped.MergeManager|: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 6, commitMemory -> 140, usedMemory ->168, mapOutput=MapOutput( AttemptIdentifier: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=4], attemptNumber=0, pathComponent=attempt_1471559811619_0002_1_00_000004_0_10009, spillType=0, spillId=-1], Type: MEMORY)\n",
      "2016-08-19 10:37:23,009 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: Completed fetch for attempt: {4, 0, attempt_1471559811619_0002_1_00_000004_0_10009} to MEMORY, csize=32, dsize=28, EndTime=1471570643009, TimeTaken=1, Rate=0.03 MB/s\n",
      "2016-08-19 10:37:23,009 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: initialmap: All inputs fetched for input vertex : initialmap\n",
      "2016-08-19 10:37:23,010 [INFO] [fetcher {initialmap} #1] |orderedgrouped.ShuffleScheduler|: copy(10 (spillsFetched=10) of 10. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)\n",
      "2016-08-19 10:37:23,011 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shutdown..fetcher {initialmap} #1, status:false, isInterrupted:false\n",
      "2016-08-19 10:37:23,011 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shutdown..fetcher {initialmap} #2, status:false, isInterrupted:false\n",
      "2016-08-19 10:37:23,011 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shutdown..fetcher {initialmap} #3, status:false, isInterrupted:false\n",
      "2016-08-19 10:37:23,011 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shutdown..fetcher {initialmap} #4, status:false, isInterrupted:false\n",
      "2016-08-19 10:37:23,012 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shutdown..fetcher {initialmap} #5, status:false, isInterrupted:false\n",
      "2016-08-19 10:37:23,012 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.ShuffleScheduler|: copy(10 (spillsFetched=10) of 10. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)\n",
      "2016-08-19 10:37:23,013 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: finalMerge called with 6 in-memory map-outputs and 4 on-disk map-outputs\n",
      "2016-08-19 10:37:23,019 [INFO] [ShuffleAndMergeRunner {initialmap}] |impl.TezMerger|: Merging 6 sorted segments\n",
      "2016-08-19 10:37:23,020 [INFO] [ShuffleAndMergeRunner {initialmap}] |impl.TezMerger|: Down to the last merge-pass, with 6 segments left of total size: 150 bytes\n",
      "2016-08-19 10:37:23,023 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Merged 6 segments, 168 bytes to disk to satisfy reduce memory limit. outputPath=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/attempt_1471559811619_0002_1_01_000000_0_10014_src_0_spill_2147483647.out.merged\n",
      "2016-08-19 10:37:23,023 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Disk file=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000001_0_10003/file.out, len=32, isLocal=true\n",
      "2016-08-19 10:37:23,023 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Disk file=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000005_0_10006/file.out, len=32, isLocal=true\n",
      "2016-08-19 10:37:23,023 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Disk file=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000006_0_10009/file.out, len=32, isLocal=true\n",
      "2016-08-19 10:37:23,023 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Disk file=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000007_0_10012/file.out, len=32, isLocal=true\n",
      "2016-08-19 10:37:23,024 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Disk file=/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/attempt_1471559811619_0002_1_01_000000_0_10014_src_0_spill_2147483647.out.merged, len=126, isLocal=false\n",
      "2016-08-19 10:37:23,024 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Merging 5 files, 254 bytes from disk\n",
      "2016-08-19 10:37:23,025 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.MergeManager|: Merging 0 segments, 0 bytes from memory into reduce\n",
      "2016-08-19 10:37:23,025 [INFO] [ShuffleAndMergeRunner {initialmap}] |impl.TezMerger|: Merging 5 sorted segments\n",
      "2016-08-19 10:37:23,027 [INFO] [ShuffleAndMergeRunner {initialmap}] |impl.TezMerger|: Down to the last merge-pass, with 5 segments left of total size: 199 bytes\n",
      "2016-08-19 10:37:23,027 [INFO] [TezChild] |reduce.ReduceProcessor|: Input is ready for consumption. Starting Output\n",
      "2016-08-19 10:37:23,028 [INFO] [TezChild] |reduce.ReduceProcessor|: Using keyClass: class org.apache.hadoop.io.BooleanWritable\n",
      "2016-08-19 10:37:23,028 [INFO] [TezChild] |reduce.ReduceProcessor|: Using valueClass: class org.apache.hadoop.io.LongWritable\n",
      "2016-08-19 10:37:23,029 [INFO] [TezChild] |reduce.ReduceProcessor|: Using comparator: org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4\n",
      "2016-08-19 10:37:23,031 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: merge complete for input vertex : initialmap\n",
      "2016-08-19 10:37:23,031 [INFO] [TezChild] |input.OrderedGroupedKVInput|: initialmap: creating ValuesIterator with comparator=org.apache.hadoop.io.BooleanWritable$Comparator, keyClass=org.apache.hadoop.io.BooleanWritable, valClass=org.apache.hadoop.io.LongWritable\n",
      "2016-08-19 10:37:23,039 [INFO] [ShuffleAndMergeRunner {initialmap}] |orderedgrouped.Shuffle|: initialmap: Shuffle Runner thread complete\n",
      "2016-08-19 10:37:23,040 [INFO] [TezChild] |input.OrderedGroupedKVInput|: initialmap: creating ValuesIterator with comparator=org.apache.hadoop.io.BooleanWritable$Comparator, keyClass=org.apache.hadoop.io.BooleanWritable, valClass=org.apache.hadoop.io.LongWritable\n",
      "2016-08-19 10:37:23,324 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_r_000000_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:23,329 [INFO] [TezChild] |processor.MRTask|: Task attempt_1471559811619_0002_r_000000_0 is allowed to commit now\n",
      "2016-08-19 10:37:23,467 [INFO] [TezChild] |output.FileOutputCommitter|: Saved output of task 'attempt_14715598116191_0002_r_000000_0' to hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/out/_temporary/1/task_14715598116191_0002_r_000000\n",
      "2016-08-19 10:37:23,467 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0\n",
      "2016-08-19 10:37:23,467 [INFO] [TezChild] |orderedgrouped.Shuffle|: Shutting down Shuffle for source: initialmap\n",
      "2016-08-19 10:37:23,467 [INFO] [TezChild] |orderedgrouped.ShuffleInputEventHandlerOrderedGrouped|: initialmap: numDmeEventsSeen=10, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose\n",
      "2016-08-19 10:37:23,468 [INFO] [TezChild] |orderedgrouped.MergeManager|: finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs\n",
      "2016-08-19 10:37:23,468 [INFO] [TezChild] |orderedgrouped.MergeManager|: Merging 0 files, 0 bytes from disk\n",
      "2016-08-19 10:37:23,468 [INFO] [TezChild] |orderedgrouped.MergeManager|: Merging 0 segments, 0 bytes from memory into reduce\n",
      "2016-08-19 10:37:23,468 [INFO] [TezChild] |impl.TezMerger|: Merging 0 sorted segments\n",
      "2016-08-19 10:37:23,469 [INFO] [TezChild] |impl.TezMerger|: Nothing to merge. Returning an empty iterator\n",
      "2016-08-19 10:37:23,469 [INFO] [TezChild] |output.MROutput|: MROutput closed\n",
      "2016-08-19 10:37:23,470 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:23,471 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_01_000000_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:23,476 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_01_000000_0: Counters: 43 [[File System Counters FILE_BYTES_READ=382, FILE_BYTES_WRITTEN=126, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=0, HDFS_BYTES_WRITTEN=215, HDFS_READ_OPS=4, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=3][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=10, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=10, GC_TIME_MILLIS=179, CPU_MILLISECONDS=1410, PHYSICAL_MEMORY_BYTES=596115456, VIRTUAL_MEMORY_BYTES=4638515200, COMMITTED_HEAP_BYTES=596115456, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=126, ADDITIONAL_SPILLS_BYTES_READ=254, SHUFFLE_BYTES=320, SHUFFLE_BYTES_DECOMPRESSED=280, SHUFFLE_BYTES_TO_MEM=192, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=128, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7627, MERGE_PHASE_TIME=7644, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=7618][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0]]\n",
      "2016-08-19 10:37:23,577 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_01_000000_0\n",
      "\n",
      "\n",
      "\n",
      "Container: container_e01_1471559811619_0002_01_000004 on testvm006_45454\n",
      "==========================================================================\n",
      "LogType:directory.info\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:14414\n",
      "Log Contents:\n",
      "ls -l:\n",
      "total 48\n",
      "-rw-r--r-- 1 yarn yarn  143 Aug 19 10:37 container_tokens\n",
      "-rwx------ 1 yarn yarn  782 Aug 19 10:37 default_container_executor.sh\n",
      "-rwx------ 1 yarn yarn  728 Aug 19 10:37 default_container_executor_session.sh\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.jar -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\n",
      "lrwxrwxrwx 1 yarn yarn  117 Aug 19 10:37 job.split -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\n",
      "lrwxrwxrwx 1 yarn yarn  125 Aug 19 10:37 job.splitmetainfo -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.xml -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\n",
      "-rwx------ 1 yarn yarn 4953 Aug 19 10:37 launch_container.sh\n",
      "lrwxrwxrwx 1 yarn yarn  119 Aug 19 10:37 tez-conf.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\n",
      "lrwxrwxrwx 1 yarn yarn   60 Aug 19 10:37 tezlib -> /hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\n",
      "drwx--x--- 2 yarn yarn 4096 Aug 19 10:37 tmp\n",
      "find -L . -maxdepth 5 -ls:\n",
      "655667    4 drwx--x---   3 yarn     yarn         4096 Aug 19 10:37 .\n",
      "655670    4 -rw-r--r--   1 yarn     yarn           12 Aug 19 10:37 ./.container_tokens.crc\n",
      "655672    4 -rw-r--r--   1 yarn     yarn           48 Aug 19 10:37 ./.launch_container.sh.crc\n",
      "655660  108 -r-x------   1 yarn     yarn       108179 Aug 19 10:37 ./job.xml\n",
      "655669    4 -rw-r--r--   1 yarn     yarn          143 Aug 19 10:37 ./container_tokens\n",
      "655675    4 -rwx------   1 yarn     yarn          782 Aug 19 10:37 ./default_container_executor.sh\n",
      "655465    4 drwxr-xr-x   3 yarn     yarn         4096 Aug 18 23:45 ./tezlib\n",
      "655485  272 -r-xr-xr-x   1 yarn     yarn       275362 Apr 25 15:39 ./tezlib/tez-mapreduce-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655488  152 -r-xr-xr-x   1 yarn     yarn       154464 Apr 25 15:39 ./tezlib/tez-tests-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655484  532 -r-xr-xr-x   1 yarn     yarn       542434 Apr 25 15:39 ./tezlib/tez-runtime-library-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655486   48 -r-xr-xr-x   1 yarn     yarn        47672 Apr 25 15:39 ./tezlib/tez-examples-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655483  164 -r-xr-xr-x   1 yarn     yarn       164267 Apr 25 15:39 ./tezlib/tez-runtime-internals-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655493    8 -r-xr-xr-x   1 yarn     yarn         6334 Apr 25 15:41 ./tezlib/tez-yarn-timeline-cache-plugin-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655491   64 -r-xr-xr-x   1 yarn     yarn        62943 Apr 25 15:41 ./tezlib/tez-history-parser-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655482   68 -r-xr-xr-x   1 yarn     yarn        67195 Apr 25 15:38 ./tezlib/tez-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655494   24 -r-xr-xr-x   1 yarn     yarn        22125 Apr 25 15:42 ./tezlib/tez-yarn-timeline-history-with-fs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655490   28 -r-xr-xr-x   1 yarn     yarn        25622 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655495    4 drwxr-xr-x   2 yarn     yarn         4096 Aug 18 23:45 ./tezlib/lib\n",
      "655534  524 -r-xr-xr-x   1 yarn     yarn       533455 Apr 23 05:17 ./tezlib/lib/protobuf-java-2.5.0.jar\n",
      "655541   20 -r-xr-xr-x   1 yarn     yarn        16560 Apr 23 08:05 ./tezlib/lib/api-asn1-api-1.0.0-M20.jar\n",
      "655548 1444 -r-xr-xr-x   1 yarn     yarn      1475955 Apr 23 06:44 ./tezlib/lib/htrace-core-3.1.0-incubating.jar\n",
      "655531  976 -r-xr-xr-x   1 yarn     yarn       995968 Apr 23 05:17 ./tezlib/lib/snappy-java-XXX.XXX.XXX.1.jar\n",
      "655554  128 -r-xr-xr-x   1 yarn     yarn       130458 Apr 23 05:45 ./tezlib/lib/jersey-client-1.9.jar\n",
      "655539  676 -r-xr-xr-x   1 yarn     yarn       691479 Apr 23 08:05 ./tezlib/lib/apacheds-kerberos-codec-2.0.0-M15.jar\n",
      "655558    8 -r-xr-xr-x   1 yarn     yarn         4467 Apr 23 05:19 ./tezlib/lib/aopalliance-1.0.jar\n",
      "655522   64 -r-xr-xr-x   1 yarn     yarn        62050 Apr 23 05:34 ./tezlib/lib/commons-logging-1.1.3.jar\n",
      "655575  152 -r-xr-xr-x   1 yarn     yarn       153798 Apr 25 14:41 ./tezlib/lib/hadoop-azure-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655503   80 -r-xr-xr-x   1 yarn     yarn        81743 Apr 23 09:18 ./tezlib/lib/jettison-1.3.4.jar\n",
      "655563  312 -r-xr-xr-x   1 yarn     yarn       319099 Apr 23 05:19 ./tezlib/lib/okhttp-2.4.0.jar\n",
      "655571  848 -r-xr-xr-x   1 yarn     yarn       865838 Apr 23 05:19 ./tezlib/lib/jackson-databind-2.2.3.jar\n",
      "655572   36 -r-xr-xr-x   1 yarn     yarn        33483 Apr 23 05:19 ./tezlib/lib/jackson-annotations-2.2.3.jar\n",
      "655551  104 -r-xr-xr-x   1 yarn     yarn       105112 Apr 23 05:17 ./tezlib/lib/servlet-api-2.5.jar\n",
      "655526  144 -r-xr-xr-x   1 yarn     yarn       143602 Apr 23 05:17 ./tezlib/lib/commons-digester-1.8.jar\n",
      "655580   36 -r-xr-xr-x   1 yarn     yarn        34835 Apr 25 14:37 ./tezlib/lib/hadoop-yarn-server-web-proxy-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655537  424 -r-xr-xr-x   1 yarn     yarn       433368 Apr 23 07:01 ./tezlib/lib/httpclient-4.2.5.jar\n",
      "655582  100 -r-xr-xr-x   1 yarn     yarn        98771 Apr 25 14:58 ./tezlib/lib/hadoop-yarn-registry-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655573 11672 -r-xr-xr-x   1 yarn     yarn     11948376 Apr 23 08:19 ./tezlib/lib/aws-java-sdk-1.7.4.jar\n",
      "655578 1500 -r-xr-xr-x   1 yarn     yarn      1533812 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-core-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655496   12 -r-xr-xr-x   1 yarn     yarn         8869 Apr 23 06:23 ./tezlib/lib/slf4j-log4j12-1.7.5.jar\n",
      "655523  572 -r-xr-xr-x   1 yarn     yarn       582965 Apr 23 05:19 ./tezlib/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar\n",
      "655579  736 -r-xr-xr-x   1 yarn     yarn       751873 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655586  700 -r-xr-xr-x   1 yarn     yarn       714194 Apr 23 05:37 ./tezlib/lib/javassist-3.18.1-GA.jar\n",
      "655535  188 -r-xr-xr-x   1 yarn     yarn       190432 Apr 23 08:06 ./tezlib/lib/gson-2.2.4.jar\n",
      "655556  696 -r-xr-xr-x   1 yarn     yarn       710492 Apr 23 05:19 ./tezlib/lib/guice-3.0.jar\n",
      "655513  148 -r-xr-xr-x   1 yarn     yarn       147952 Apr 23 05:44 ./tezlib/lib/jersey-json-1.9.jar\n",
      "655576  652 -r-xr-xr-x   1 yarn     yarn       666763 Apr 23 05:19 ./tezlib/lib/azure-storage-2.2.0.jar\n",
      "655560  184 -r-xr-xr-x   1 yarn     yarn       185401 Apr 25 15:01 ./tezlib/lib/hadoop-yarn-client-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655569  532 -r-xr-xr-x   1 yarn     yarn       540751 Apr 23 05:19 ./tezlib/lib/jetty-6.1.26.hwx.jar\n",
      "655509   16 -r-xr-xr-x   1 yarn     yarn        15010 Apr 23 05:17 ./tezlib/lib/xmlenc-0.52.jar\n",
      "655585  336 -r-xr-xr-x   1 yarn     yarn       340884 Apr 23 05:19 ./tezlib/lib/fst-2.24.jar\n",
      "655550 1672 -r-xr-xr-x   1 yarn     yarn      1710975 Apr 25 14:57 ./tezlib/lib/hadoop-yarn-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655512  576 -r-xr-xr-x   1 yarn     yarn       588337 Apr 23 05:37 ./tezlib/lib/commons-collections-3.2.2.jar\n",
      "655533   96 -r-xr-xr-x   1 yarn     yarn        94672 Apr 23 05:17 ./tezlib/lib/xz-1.0.jar\n",
      "655547  268 -r-xr-xr-x   1 yarn     yarn       270342 Apr 23 06:44 ./tezlib/lib/curator-recipes-2.7.1.jar\n",
      "655561  736 -r-xr-xr-x   1 yarn     yarn       751238 Apr 23 09:18 ./tezlib/lib/commons-collections4-4.1.jar\n",
      "655508 1564 -r-xr-xr-x   1 yarn     yarn      1599627 Apr 23 06:43 ./tezlib/lib/commons-math3-3.1.1.jar\n",
      "655501  184 -r-xr-xr-x   1 yarn     yarn       185140 Apr 23 05:19 ./tezlib/lib/commons-io-2.4.jar\n",
      "655518  228 -r-xr-xr-x   1 yarn     yarn       232248 Apr 23 05:28 ./tezlib/lib/jackson-core-asl-1.9.13.jar\n",
      "655536   72 -r-xr-xr-x   1 yarn     yarn        71570 Apr 25 14:27 ./tezlib/lib/hadoop-auth-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655505 3420 -r-xr-xr-x   1 yarn     yarn      3498444 Apr 25 14:46 ./tezlib/lib/hadoop-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655497   28 -r-xr-xr-x   1 yarn     yarn        26084 Apr 23 05:34 ./tezlib/lib/slf4j-api-1.7.5.jar\n",
      "655540   44 -r-xr-xr-x   1 yarn     yarn        44925 Apr 23 08:05 ./tezlib/lib/apacheds-i18n-2.0.0-M15.jar\n",
      "655549 2060 -r-xr-xr-x   1 yarn     yarn      2108699 Apr 25 14:55 ./tezlib/lib/hadoop-yarn-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655527  188 -r-xr-xr-x   1 yarn     yarn       188671 Apr 23 05:14 ./tezlib/lib/commons-beanutils-1.7.0.jar\n",
      "655507   44 -r-xr-xr-x   1 yarn     yarn        41123 Apr 23 05:14 ./tezlib/lib/commons-cli-1.2.jar\n",
      "655516   24 -r-xr-xr-x   1 yarn     yarn        23346 Apr 23 05:14 ./tezlib/lib/stax-api-1.0-2.jar\n",
      "655562 8228 -r-xr-xr-x   1 yarn     yarn      8421957 Apr 25 14:50 ./tezlib/lib/hadoop-hdfs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655544 1172 -r-xr-xr-x   1 yarn     yarn      1199572 Apr 23 05:47 ./tezlib/lib/netty-3.6.2.Final.jar\n",
      "655515  104 -r-xr-xr-x   1 yarn     yarn       105134 Apr 23 05:17 ./tezlib/lib/jaxb-api-2.2.2.jar\n",
      "655545  184 -r-xr-xr-x   1 yarn     yarn       186273 Apr 23 06:43 ./tezlib/lib/curator-framework-2.7.1.jar\n",
      "655524  192 -r-xr-xr-x   1 yarn     yarn       192699 Apr 23 05:19 ./tezlib/lib/jackson-core-2.2.3.jar\n",
      "655538  224 -r-xr-xr-x   1 yarn     yarn       227275 Apr 23 07:14 ./tezlib/lib/httpcore-4.2.4.jar\n",
      "655568 1024 -r-xr-xr-x   1 yarn     yarn      1045744 Apr 23 05:19 ./tezlib/lib/leveldbjni-all-1.8.jar\n",
      "655500   36 -r-xr-xr-x   1 yarn     yarn        33031 Apr 23 09:18 ./tezlib/lib/jsr305-2.0.3.jar\n",
      "655525  292 -r-xr-xr-x   1 yarn     yarn       298829 Apr 23 05:17 ./tezlib/lib/commons-configuration-1.6.jar\n",
      "655570  104 -r-xr-xr-x   1 yarn     yarn       104578 Apr 25 14:41 ./tezlib/lib/hadoop-aws-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655574  616 -r-xr-xr-x   1 yarn     yarn       627814 Apr 23 08:19 ./tezlib/lib/joda-time-2.9.3.jar\n",
      "655511  268 -r-xr-xr-x   1 yarn     yarn       273370 Apr 23 05:17 ./tezlib/lib/commons-net-3.1.jar\n",
      "655532  236 -r-xr-xr-x   1 yarn     yarn       241367 Apr 23 06:43 ./tezlib/lib/commons-compress-1.4.1.jar\n",
      "655506   20 -r-xr-xr-x   1 yarn     yarn        17371 Apr 25 14:27 ./tezlib/lib/hadoop-annotations-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655546   68 -r-xr-xr-x   1 yarn     yarn        69500 Apr 23 06:43 ./tezlib/lib/curator-client-2.7.1.jar\n",
      "655557    4 -r-xr-xr-x   1 yarn     yarn         2497 Apr 23 05:19 ./tezlib/lib/javax.inject-1.jar\n",
      "655553  448 -r-xr-xr-x   1 yarn     yarn       458739 Apr 23 05:45 ./tezlib/lib/jersey-core-1.9.jar\n",
      "655499 1612 -r-xr-xr-x   1 yarn     yarn      1648200 Apr 23 05:18 ./tezlib/lib/guava-11.0.2.jar\n",
      "655504   28 -r-xr-xr-x   1 yarn     yarn        26514 Apr 23 05:19 ./tezlib/lib/stax-api-1.0.1.jar\n",
      "655577  404 -r-xr-xr-x   1 yarn     yarn       412739 Apr 23 06:23 ./tezlib/lib/commons-lang3-3.3.2.jar\n",
      "655543  776 -r-xr-xr-x   1 yarn     yarn       794058 Apr 25 14:27 ./tezlib/lib/zookeeper-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655529  300 -r-xr-xr-x   1 yarn     yarn       303139 Apr 25 14:28 ./tezlib/lib/avro-1.7.4.jar\n",
      "655555   64 -r-xr-xr-x   1 yarn     yarn        65012 Apr 23 05:19 ./tezlib/lib/guice-servlet-3.0.jar\n",
      "655564   64 -r-xr-xr-x   1 yarn     yarn        64661 Apr 23 05:19 ./tezlib/lib/okio-1.4.0.jar\n",
      "655567  192 -r-xr-xr-x   1 yarn     yarn       194354 Apr 23 05:14 ./tezlib/lib/xml-apis-1.3.04.jar\n",
      "655565 1740 -r-xr-xr-x   1 yarn     yarn      1779991 Apr 23 05:19 ./tezlib/lib/netty-all-4.0.23.Final.jar\n",
      "655510   60 -r-xr-xr-x   1 yarn     yarn        58160 Apr 23 08:05 ./tezlib/lib/commons-codec-1.4.jar\n",
      "655588   60 -r-xr-xr-x   1 yarn     yarn        60819 Apr 25 14:38 ./tezlib/lib/hadoop-yarn-server-timeline-plugins-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655587   44 -r-xr-xr-x   1 yarn     yarn        41755 Apr 23 05:51 ./tezlib/lib/objenesis-2.1.jar\n",
      "655584  244 -r-xr-xr-x   1 yarn     yarn       246782 Apr 25 15:00 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655520   20 -r-xr-xr-x   1 yarn     yarn        18336 Apr 23 08:06 ./tezlib/lib/jackson-jaxrs-1.9.13.jar\n",
      "655519  764 -r-xr-xr-x   1 yarn     yarn       780664 Apr 23 05:17 ./tezlib/lib/jackson-mapper-asl-1.9.13.jar\n",
      "655581  408 -r-xr-xr-x   1 yarn     yarn       413811 Apr 25 14:59 ./tezlib/lib/hadoop-yarn-server-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655552  176 -r-xr-xr-x   1 yarn     yarn       177487 Apr 23 05:19 ./tezlib/lib/jetty-util-6.1.26.hwx.jar\n",
      "655528  204 -r-xr-xr-x   1 yarn     yarn       206035 Apr 23 05:17 ./tezlib/lib/commons-beanutils-core-1.8.0.jar\n",
      "655559   16 -r-xr-xr-x   1 yarn     yarn        14786 Apr 23 05:45 ./tezlib/lib/jersey-guice-1.9.jar\n",
      "655514  872 -r-xr-xr-x   1 yarn     yarn       890168 Apr 23 05:17 ./tezlib/lib/jaxb-impl-2.2.3-1.jar\n",
      "655521   28 -r-xr-xr-x   1 yarn     yarn        27084 Apr 23 05:17 ./tezlib/lib/jackson-xc-1.9.13.jar\n",
      "655583  132 -r-xr-xr-x   1 yarn     yarn       134133 Apr 23 08:04 ./tezlib/lib/servlet-api-2.5-20081211.jar\n",
      "655502  280 -r-xr-xr-x   1 yarn     yarn       284220 Apr 23 05:14 ./tezlib/lib/commons-lang-2.6.jar\n",
      "655498  480 -r-xr-xr-x   1 yarn     yarn       489884 Apr 23 05:17 ./tezlib/lib/log4j-1.2.17.jar\n",
      "655530   32 -r-xr-xr-x   1 yarn     yarn        29555 Apr 23 05:17 ./tezlib/lib/paranamer-2.3.jar\n",
      "655566 1204 -r-xr-xr-x   1 yarn     yarn      1229125 Apr 23 05:14 ./tezlib/lib/xercesImpl-2.9.1.jar\n",
      "655517   64 -r-xr-xr-x   1 yarn     yarn        62983 Apr 23 05:17 ./tezlib/lib/activation-1.1.jar\n",
      "655542   80 -r-xr-xr-x   1 yarn     yarn        79912 Apr 23 08:05 ./tezlib/lib/api-util-1.0.0-M20.jar\n",
      "655492    8 -r-xr-xr-x   1 yarn     yarn         7784 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-with-acls-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655481  868 -r-xr-xr-x   1 yarn     yarn       888761 Apr 25 15:38 ./tezlib/tez-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655487 1196 -r-xr-xr-x   1 yarn     yarn      1222496 Apr 25 15:39 ./tezlib/tez-dag-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655489 1600 -r-xr-xr-x   1 yarn     yarn      1636310 Apr 25 15:40 ./tezlib/tez-ui-XXX.XXX.XXX.2.4.2.0-258.war\n",
      "655674    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor_session.sh.crc\n",
      "655651    4 -r-x------   1 yarn     yarn         1467 Aug 19 10:37 ./job.split\n",
      "655671    8 -rwx------   1 yarn     yarn         4953 Aug 19 10:37 ./launch_container.sh\n",
      "655673    4 -rwx------   1 yarn     yarn          728 Aug 19 10:37 ./default_container_executor_session.sh\n",
      "655657    4 -r-x------   1 yarn     yarn          357 Aug 19 10:37 ./job.splitmetainfo\n",
      "655654   44 -r-x------   1 yarn     yarn        44350 Aug 19 10:37 ./tez-conf.pb\n",
      "655676    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor.sh.crc\n",
      "655668    4 drwx--x---   2 yarn     yarn         4096 Aug 19 10:37 ./tmp\n",
      "655648  268 -r-x------   1 yarn     yarn       273458 Aug 19 10:37 ./job.jar\n",
      "broken symlinks(find -L . -maxdepth 5 -type l -ls):\n",
      "End of LogType:directory.info\n",
      "\n",
      "LogType:launch_container.sh\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:4953\n",
      "Log Contents:\n",
      "#!/bin/bash\n",
      "\n",
      "export SHELL=\"/bin/bash\"\n",
      "export NM_HTTP_PORT=\"8042\"\n",
      "export LOCAL_DIRS=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002\"\n",
      "export JAVA_HOME=\"/usr/java/jdk1.7.0_75\"\n",
      "export NM_AUX_SERVICE_mapreduce_shuffle=\"AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n",
      "\"\n",
      "export HADOOP_YARN_HOME=\"/usr/hdp/XXX.XXX.XXX.0-258/hadoop-yarn\"\n",
      "export CLASSPATH=\"$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:\"\n",
      "export HADOOP_TOKEN_FILE_LOCATION=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/container_tokens\"\n",
      "export NM_HOST=\"testvm006\"\n",
      "export HADOOP_ROOT_LOGGER=\"INFO,CLA\"\n",
      "export JVM_PID=\"$$\"\n",
      "export USER=\"ansible\"\n",
      "export PWD=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004\"\n",
      "export CONTAINER_ID=\"container_e01_1471559811619_0002_01_000004\"\n",
      "export HOME=\"/home/\"\n",
      "export NM_PORT=\"45454\"\n",
      "export LOGNAME=\"ansible\"\n",
      "export HADOOP_CONF_DIR=\"/usr/hdp/current/hadoop-yarn-nodemanager/../hadoop/conf\"\n",
      "export MALLOC_ARENA_MAX=\"4\"\n",
      "export LD_LIBRARY_PATH=\"$PWD:$HADOOP_COMMON_HOME/lib/native:$PWD\"\n",
      "export LOG_DIRS=\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004\"\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\" \"tezlib\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\" \"job.split\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\" \"job.xml\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\" \"tez-conf.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\" \"job.splitmetainfo\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\" \"job.jar\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "# Creating copy of launch script\n",
      "cp \"launch_container.sh\" \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/launch_container.sh\"\n",
      "chmod 640 \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/launch_container.sh\"\n",
      "# Determining directory contents\n",
      "echo \"ls -l:\" 1>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "ls -l 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "echo \"find -L . -maxdepth 5 -ls:\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "find -L . -maxdepth 5 -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "echo \"broken symlinks(find -L . -maxdepth 5 -type l -ls):\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "find -L . -maxdepth 5 -type l -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/directory.info\"\n",
      "exec /bin/bash -c \"$JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3840m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004 -Dtez.root.logger=INFO,CLA  -Djava.io.tmpdir=$PWD/tmp org.apache.tez.runtime.task.TezChild XXX.XXX.XXX.73 36714 container_e01_1471559811619_0002_01_000004 application_1471559811619_0002 1 1>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/stdout 2>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/stderr \"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "End of LogType:launch_container.sh\n",
      "\n",
      "LogType:stderr\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:564\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:17 Completed running task attempt: attempt_1471559811619_0002_1_00_000009_0\n",
      "End of LogType:stderr\n",
      "\n",
      "LogType:stdout\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:564\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14 Completed running task attempt: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:17 Completed running task attempt: attempt_1471559811619_0002_1_00_000009_0\n",
      "End of LogType:stdout\n",
      "\n",
      "LogType:syslog\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:2157\n",
      "Log Contents:\n",
      "2016-08-19 10:37:09,134 [INFO] [main] |task.TezChild|: TezChild starting with PID=18142, containerIdentifier=container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:10,445 [WARN] [main] |util.NativeCodeLoader|: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "2016-08-19 10:37:11,165 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:11,653 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:11,656 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14,100 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:14,103 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,114 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,362 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000004\n",
      "2016-08-19 10:37:14,367 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:14,368 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:18,105 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000004\n",
      "End of LogType:syslog\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000002_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:8586\n",
      "Log Contents:\n",
      "2016-08-19 10:37:11,664 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=1\n",
      "2016-08-19 10:37:11,693 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:11,693 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000002_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:11,696 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:11,725 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:11,946 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000002_0\n",
      "2016-08-19 10:37:11,982 [INFO] [I/O Setup 0 Initialize: {MRInput}] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120\n",
      "2016-08-19 10:37:11,990 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:11,990 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:12,475 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:12,475 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:12,476 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:12,476 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:12,484 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:12,670 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:12,678 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:12,679 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:12,679 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:12,679 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:12,681 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: MRInput being auto started by the framework. Subsequent instances will not be auto-started\n",
      "2016-08-19 10:37:12,681 [INFO] [I/O Setup 1 Start: {MRInput}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: MRInput\n",
      "2016-08-19 10:37:12,681 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1\n",
      "2016-08-19 10:37:12,682 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start\n",
      "2016-08-19 10:37:12,682 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:12,685 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:12,686 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000002_0_10001\n",
      "2016-08-19 10:37:12,750 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@3ebfe943, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:12,757 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:12,805 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:12,859 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:12,903 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/job.split\n",
      "2016-08-19 10:37:13,997 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:13,999 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part2:0+118\n",
      "2016-08-19 10:37:14,004 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000002_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:14,004 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0\n",
      "2016-08-19 10:37:14,009 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:14,009 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:14,009 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:14,014 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:14,014 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000002_0_10003_0/file.out\n",
      "2016-08-19 10:37:14,035 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:14,060 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000002_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:14,091 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000002_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1557, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=208, CPU_MILLISECONDS=3400, PHYSICAL_MEMORY_BYTES=290455552, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=290455552, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:14,100 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000002_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000008_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:7805\n",
      "Log Contents:\n",
      "2016-08-19 10:37:14,115 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:14,115 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000008_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:14,116 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:14,116 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,149 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000008_0\n",
      "2016-08-19 10:37:14,150 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:14,150 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:14,156 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:14,156 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:14,157 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:14,158 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:14,158 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:14,158 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:14,158 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:14,159 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:14,159 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:14,159 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:14,160 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:14,160 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:14,165 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,165 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000008_0_10004\n",
      "2016-08-19 10:37:14,170 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@5188245d, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:14,170 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:14,211 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:14,214 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:14,218 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/job.split\n",
      "2016-08-19 10:37:14,250 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:14,250 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part8:0+118\n",
      "2016-08-19 10:37:14,251 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000008_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:14,251 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0\n",
      "2016-08-19 10:37:14,251 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:14,252 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:14,252 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:14,253 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:14,253 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000008_0_10006_0/file.out\n",
      "2016-08-19 10:37:14,255 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:14,256 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000008_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:14,261 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000008_0: Counters: 26 [[File System Counters FILE_BYTES_READ=681, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=200, PHYSICAL_MEMORY_BYTES=395837440, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=395837440, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:14,362 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000008_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000009_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:7806\n",
      "Log Contents:\n",
      "2016-08-19 10:37:14,369 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:14,369 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000009_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:14,369 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:14,370 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:14,388 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000009_0\n",
      "2016-08-19 10:37:14,389 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:14,389 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:14,395 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:14,395 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:14,395 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:14,396 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:14,397 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:14,397 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000009_0_10007\n",
      "2016-08-19 10:37:14,401 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@714ed0b9, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@296b4de4, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:14,402 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:17,924 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:17,927 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:17,935 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000004/job.split\n",
      "2016-08-19 10:37:17,988 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:17,988 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part9:0+118\n",
      "2016-08-19 10:37:17,990 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000009_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:17,991 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0\n",
      "2016-08-19 10:37:17,991 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:17,991 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:17,991 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:17,993 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:17,993 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000009_0_10009_0/file.out\n",
      "2016-08-19 10:37:17,997 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:17,998 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000009_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:18,003 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000009_0: Counters: 26 [[File System Counters FILE_BYTES_READ=535, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=3200, PHYSICAL_MEMORY_BYTES=501219328, VIRTUAL_MEMORY_BYTES=4624896000, COMMITTED_HEAP_BYTES=501219328, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:18,104 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000009_0\n",
      "\n",
      "\n",
      "\n",
      "Container: container_e01_1471559811619_0002_01_000002 on testvm007_45454\n",
      "==========================================================================\n",
      "LogType:directory.info\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:14414\n",
      "Log Contents:\n",
      "ls -l:\n",
      "total 48\n",
      "-rw-r--r-- 1 yarn yarn  143 Aug 19 10:37 container_tokens\n",
      "-rwx------ 1 yarn yarn  782 Aug 19 10:37 default_container_executor.sh\n",
      "-rwx------ 1 yarn yarn  728 Aug 19 10:37 default_container_executor_session.sh\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.jar -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\n",
      "lrwxrwxrwx 1 yarn yarn  117 Aug 19 10:37 job.split -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\n",
      "lrwxrwxrwx 1 yarn yarn  125 Aug 19 10:37 job.splitmetainfo -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\n",
      "lrwxrwxrwx 1 yarn yarn  115 Aug 19 10:37 job.xml -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\n",
      "-rwx------ 1 yarn yarn 4953 Aug 19 10:37 launch_container.sh\n",
      "lrwxrwxrwx 1 yarn yarn  119 Aug 19 10:37 tez-conf.pb -> /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\n",
      "lrwxrwxrwx 1 yarn yarn   60 Aug 19 10:37 tezlib -> /hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\n",
      "drwx--x--- 2 yarn yarn 4096 Aug 19 10:37 tmp\n",
      "find -L . -maxdepth 5 -ls:\n",
      "655673    4 drwx--x---   3 yarn     yarn         4096 Aug 19 10:37 .\n",
      "655676    4 -rw-r--r--   1 yarn     yarn           12 Aug 19 10:37 ./.container_tokens.crc\n",
      "655678    4 -rw-r--r--   1 yarn     yarn           48 Aug 19 10:37 ./.launch_container.sh.crc\n",
      "655666  108 -r-x------   1 yarn     yarn       108179 Aug 19 10:37 ./job.xml\n",
      "655675    4 -rw-r--r--   1 yarn     yarn          143 Aug 19 10:37 ./container_tokens\n",
      "655681    4 -rwx------   1 yarn     yarn          782 Aug 19 10:37 ./default_container_executor.sh\n",
      "655452    4 drwxr-xr-x   3 yarn     yarn         4096 Aug 18 23:45 ./tezlib\n",
      "655475  272 -r-xr-xr-x   1 yarn     yarn       275362 Apr 25 15:39 ./tezlib/tez-mapreduce-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655478  152 -r-xr-xr-x   1 yarn     yarn       154464 Apr 25 15:39 ./tezlib/tez-tests-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655474  532 -r-xr-xr-x   1 yarn     yarn       542434 Apr 25 15:39 ./tezlib/tez-runtime-library-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655476   48 -r-xr-xr-x   1 yarn     yarn        47672 Apr 25 15:39 ./tezlib/tez-examples-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655473  164 -r-xr-xr-x   1 yarn     yarn       164267 Apr 25 15:39 ./tezlib/tez-runtime-internals-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655483    8 -r-xr-xr-x   1 yarn     yarn         6334 Apr 25 15:41 ./tezlib/tez-yarn-timeline-cache-plugin-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655481   64 -r-xr-xr-x   1 yarn     yarn        62943 Apr 25 15:41 ./tezlib/tez-history-parser-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655472   68 -r-xr-xr-x   1 yarn     yarn        67195 Apr 25 15:38 ./tezlib/tez-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655484   24 -r-xr-xr-x   1 yarn     yarn        22125 Apr 25 15:42 ./tezlib/tez-yarn-timeline-history-with-fs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655480   28 -r-xr-xr-x   1 yarn     yarn        25622 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655485    4 drwxr-xr-x   2 yarn     yarn         4096 Aug 18 23:45 ./tezlib/lib\n",
      "655524  524 -r-xr-xr-x   1 yarn     yarn       533455 Apr 23 05:17 ./tezlib/lib/protobuf-java-2.5.0.jar\n",
      "655531   20 -r-xr-xr-x   1 yarn     yarn        16560 Apr 23 08:05 ./tezlib/lib/api-asn1-api-1.0.0-M20.jar\n",
      "655538 1444 -r-xr-xr-x   1 yarn     yarn      1475955 Apr 23 06:44 ./tezlib/lib/htrace-core-3.1.0-incubating.jar\n",
      "655521  976 -r-xr-xr-x   1 yarn     yarn       995968 Apr 23 05:17 ./tezlib/lib/snappy-java-XXX.XXX.XXX.1.jar\n",
      "655544  128 -r-xr-xr-x   1 yarn     yarn       130458 Apr 23 05:45 ./tezlib/lib/jersey-client-1.9.jar\n",
      "655529  676 -r-xr-xr-x   1 yarn     yarn       691479 Apr 23 08:05 ./tezlib/lib/apacheds-kerberos-codec-2.0.0-M15.jar\n",
      "655548    8 -r-xr-xr-x   1 yarn     yarn         4467 Apr 23 05:19 ./tezlib/lib/aopalliance-1.0.jar\n",
      "655512   64 -r-xr-xr-x   1 yarn     yarn        62050 Apr 23 05:34 ./tezlib/lib/commons-logging-1.1.3.jar\n",
      "655565  152 -r-xr-xr-x   1 yarn     yarn       153798 Apr 25 14:41 ./tezlib/lib/hadoop-azure-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655493   80 -r-xr-xr-x   1 yarn     yarn        81743 Apr 23 09:18 ./tezlib/lib/jettison-1.3.4.jar\n",
      "655553  312 -r-xr-xr-x   1 yarn     yarn       319099 Apr 23 05:19 ./tezlib/lib/okhttp-2.4.0.jar\n",
      "655561  848 -r-xr-xr-x   1 yarn     yarn       865838 Apr 23 05:19 ./tezlib/lib/jackson-databind-2.2.3.jar\n",
      "655562   36 -r-xr-xr-x   1 yarn     yarn        33483 Apr 23 05:19 ./tezlib/lib/jackson-annotations-2.2.3.jar\n",
      "655541  104 -r-xr-xr-x   1 yarn     yarn       105112 Apr 23 05:17 ./tezlib/lib/servlet-api-2.5.jar\n",
      "655516  144 -r-xr-xr-x   1 yarn     yarn       143602 Apr 23 05:17 ./tezlib/lib/commons-digester-1.8.jar\n",
      "655570   36 -r-xr-xr-x   1 yarn     yarn        34835 Apr 25 14:37 ./tezlib/lib/hadoop-yarn-server-web-proxy-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655527  424 -r-xr-xr-x   1 yarn     yarn       433368 Apr 23 07:01 ./tezlib/lib/httpclient-4.2.5.jar\n",
      "655572  100 -r-xr-xr-x   1 yarn     yarn        98771 Apr 25 14:58 ./tezlib/lib/hadoop-yarn-registry-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655563 11672 -r-xr-xr-x   1 yarn     yarn     11948376 Apr 23 08:19 ./tezlib/lib/aws-java-sdk-1.7.4.jar\n",
      "655568 1500 -r-xr-xr-x   1 yarn     yarn      1533812 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-core-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655486   12 -r-xr-xr-x   1 yarn     yarn         8869 Apr 23 06:23 ./tezlib/lib/slf4j-log4j12-1.7.5.jar\n",
      "655513  572 -r-xr-xr-x   1 yarn     yarn       582965 Apr 23 05:19 ./tezlib/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar\n",
      "655569  736 -r-xr-xr-x   1 yarn     yarn       751873 Apr 25 15:03 ./tezlib/lib/hadoop-mapreduce-client-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655576  700 -r-xr-xr-x   1 yarn     yarn       714194 Apr 23 05:37 ./tezlib/lib/javassist-3.18.1-GA.jar\n",
      "655525  188 -r-xr-xr-x   1 yarn     yarn       190432 Apr 23 08:06 ./tezlib/lib/gson-2.2.4.jar\n",
      "655546  696 -r-xr-xr-x   1 yarn     yarn       710492 Apr 23 05:19 ./tezlib/lib/guice-3.0.jar\n",
      "655503  148 -r-xr-xr-x   1 yarn     yarn       147952 Apr 23 05:44 ./tezlib/lib/jersey-json-1.9.jar\n",
      "655566  652 -r-xr-xr-x   1 yarn     yarn       666763 Apr 23 05:19 ./tezlib/lib/azure-storage-2.2.0.jar\n",
      "655550  184 -r-xr-xr-x   1 yarn     yarn       185401 Apr 25 15:01 ./tezlib/lib/hadoop-yarn-client-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655559  532 -r-xr-xr-x   1 yarn     yarn       540751 Apr 23 05:19 ./tezlib/lib/jetty-6.1.26.hwx.jar\n",
      "655499   16 -r-xr-xr-x   1 yarn     yarn        15010 Apr 23 05:17 ./tezlib/lib/xmlenc-0.52.jar\n",
      "655575  336 -r-xr-xr-x   1 yarn     yarn       340884 Apr 23 05:19 ./tezlib/lib/fst-2.24.jar\n",
      "655540 1672 -r-xr-xr-x   1 yarn     yarn      1710975 Apr 25 14:57 ./tezlib/lib/hadoop-yarn-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655502  576 -r-xr-xr-x   1 yarn     yarn       588337 Apr 23 05:37 ./tezlib/lib/commons-collections-3.2.2.jar\n",
      "655523   96 -r-xr-xr-x   1 yarn     yarn        94672 Apr 23 05:17 ./tezlib/lib/xz-1.0.jar\n",
      "655537  268 -r-xr-xr-x   1 yarn     yarn       270342 Apr 23 06:44 ./tezlib/lib/curator-recipes-2.7.1.jar\n",
      "655551  736 -r-xr-xr-x   1 yarn     yarn       751238 Apr 23 09:18 ./tezlib/lib/commons-collections4-4.1.jar\n",
      "655498 1564 -r-xr-xr-x   1 yarn     yarn      1599627 Apr 23 06:43 ./tezlib/lib/commons-math3-3.1.1.jar\n",
      "655491  184 -r-xr-xr-x   1 yarn     yarn       185140 Apr 23 05:19 ./tezlib/lib/commons-io-2.4.jar\n",
      "655508  228 -r-xr-xr-x   1 yarn     yarn       232248 Apr 23 05:28 ./tezlib/lib/jackson-core-asl-1.9.13.jar\n",
      "655526   72 -r-xr-xr-x   1 yarn     yarn        71570 Apr 25 14:27 ./tezlib/lib/hadoop-auth-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655495 3420 -r-xr-xr-x   1 yarn     yarn      3498444 Apr 25 14:46 ./tezlib/lib/hadoop-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655487   28 -r-xr-xr-x   1 yarn     yarn        26084 Apr 23 05:34 ./tezlib/lib/slf4j-api-1.7.5.jar\n",
      "655530   44 -r-xr-xr-x   1 yarn     yarn        44925 Apr 23 08:05 ./tezlib/lib/apacheds-i18n-2.0.0-M15.jar\n",
      "655539 2060 -r-xr-xr-x   1 yarn     yarn      2108699 Apr 25 14:55 ./tezlib/lib/hadoop-yarn-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655517  188 -r-xr-xr-x   1 yarn     yarn       188671 Apr 23 05:14 ./tezlib/lib/commons-beanutils-1.7.0.jar\n",
      "655497   44 -r-xr-xr-x   1 yarn     yarn        41123 Apr 23 05:14 ./tezlib/lib/commons-cli-1.2.jar\n",
      "655506   24 -r-xr-xr-x   1 yarn     yarn        23346 Apr 23 05:14 ./tezlib/lib/stax-api-1.0-2.jar\n",
      "655552 8228 -r-xr-xr-x   1 yarn     yarn      8421957 Apr 25 14:50 ./tezlib/lib/hadoop-hdfs-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655534 1172 -r-xr-xr-x   1 yarn     yarn      1199572 Apr 23 05:47 ./tezlib/lib/netty-3.6.2.Final.jar\n",
      "655505  104 -r-xr-xr-x   1 yarn     yarn       105134 Apr 23 05:17 ./tezlib/lib/jaxb-api-2.2.2.jar\n",
      "655535  184 -r-xr-xr-x   1 yarn     yarn       186273 Apr 23 06:43 ./tezlib/lib/curator-framework-2.7.1.jar\n",
      "655514  192 -r-xr-xr-x   1 yarn     yarn       192699 Apr 23 05:19 ./tezlib/lib/jackson-core-2.2.3.jar\n",
      "655528  224 -r-xr-xr-x   1 yarn     yarn       227275 Apr 23 07:14 ./tezlib/lib/httpcore-4.2.4.jar\n",
      "655558 1024 -r-xr-xr-x   1 yarn     yarn      1045744 Apr 23 05:19 ./tezlib/lib/leveldbjni-all-1.8.jar\n",
      "655490   36 -r-xr-xr-x   1 yarn     yarn        33031 Apr 23 09:18 ./tezlib/lib/jsr305-2.0.3.jar\n",
      "655515  292 -r-xr-xr-x   1 yarn     yarn       298829 Apr 23 05:17 ./tezlib/lib/commons-configuration-1.6.jar\n",
      "655560  104 -r-xr-xr-x   1 yarn     yarn       104578 Apr 25 14:41 ./tezlib/lib/hadoop-aws-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655564  616 -r-xr-xr-x   1 yarn     yarn       627814 Apr 23 08:19 ./tezlib/lib/joda-time-2.9.3.jar\n",
      "655501  268 -r-xr-xr-x   1 yarn     yarn       273370 Apr 23 05:17 ./tezlib/lib/commons-net-3.1.jar\n",
      "655522  236 -r-xr-xr-x   1 yarn     yarn       241367 Apr 23 06:43 ./tezlib/lib/commons-compress-1.4.1.jar\n",
      "655496   20 -r-xr-xr-x   1 yarn     yarn        17371 Apr 25 14:27 ./tezlib/lib/hadoop-annotations-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655536   68 -r-xr-xr-x   1 yarn     yarn        69500 Apr 23 06:43 ./tezlib/lib/curator-client-2.7.1.jar\n",
      "655547    4 -r-xr-xr-x   1 yarn     yarn         2497 Apr 23 05:19 ./tezlib/lib/javax.inject-1.jar\n",
      "655543  448 -r-xr-xr-x   1 yarn     yarn       458739 Apr 23 05:45 ./tezlib/lib/jersey-core-1.9.jar\n",
      "655489 1612 -r-xr-xr-x   1 yarn     yarn      1648200 Apr 23 05:18 ./tezlib/lib/guava-11.0.2.jar\n",
      "655494   28 -r-xr-xr-x   1 yarn     yarn        26514 Apr 23 05:19 ./tezlib/lib/stax-api-1.0.1.jar\n",
      "655567  404 -r-xr-xr-x   1 yarn     yarn       412739 Apr 23 06:23 ./tezlib/lib/commons-lang3-3.3.2.jar\n",
      "655533  776 -r-xr-xr-x   1 yarn     yarn       794058 Apr 25 14:27 ./tezlib/lib/zookeeper-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655519  300 -r-xr-xr-x   1 yarn     yarn       303139 Apr 25 14:28 ./tezlib/lib/avro-1.7.4.jar\n",
      "655545   64 -r-xr-xr-x   1 yarn     yarn        65012 Apr 23 05:19 ./tezlib/lib/guice-servlet-3.0.jar\n",
      "655554   64 -r-xr-xr-x   1 yarn     yarn        64661 Apr 23 05:19 ./tezlib/lib/okio-1.4.0.jar\n",
      "655557  192 -r-xr-xr-x   1 yarn     yarn       194354 Apr 23 05:14 ./tezlib/lib/xml-apis-1.3.04.jar\n",
      "655555 1740 -r-xr-xr-x   1 yarn     yarn      1779991 Apr 23 05:19 ./tezlib/lib/netty-all-4.0.23.Final.jar\n",
      "655500   60 -r-xr-xr-x   1 yarn     yarn        58160 Apr 23 08:05 ./tezlib/lib/commons-codec-1.4.jar\n",
      "655578   60 -r-xr-xr-x   1 yarn     yarn        60819 Apr 25 14:38 ./tezlib/lib/hadoop-yarn-server-timeline-plugins-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655577   44 -r-xr-xr-x   1 yarn     yarn        41755 Apr 23 05:51 ./tezlib/lib/objenesis-2.1.jar\n",
      "655574  244 -r-xr-xr-x   1 yarn     yarn       246782 Apr 25 15:00 ./tezlib/lib/hadoop-yarn-server-applicationhistoryservice-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655510   20 -r-xr-xr-x   1 yarn     yarn        18336 Apr 23 08:06 ./tezlib/lib/jackson-jaxrs-1.9.13.jar\n",
      "655509  764 -r-xr-xr-x   1 yarn     yarn       780664 Apr 23 05:17 ./tezlib/lib/jackson-mapper-asl-1.9.13.jar\n",
      "655571  408 -r-xr-xr-x   1 yarn     yarn       413811 Apr 25 14:59 ./tezlib/lib/hadoop-yarn-server-common-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655542  176 -r-xr-xr-x   1 yarn     yarn       177487 Apr 23 05:19 ./tezlib/lib/jetty-util-6.1.26.hwx.jar\n",
      "655518  204 -r-xr-xr-x   1 yarn     yarn       206035 Apr 23 05:17 ./tezlib/lib/commons-beanutils-core-1.8.0.jar\n",
      "655549   16 -r-xr-xr-x   1 yarn     yarn        14786 Apr 23 05:45 ./tezlib/lib/jersey-guice-1.9.jar\n",
      "655504  872 -r-xr-xr-x   1 yarn     yarn       890168 Apr 23 05:17 ./tezlib/lib/jaxb-impl-2.2.3-1.jar\n",
      "655511   28 -r-xr-xr-x   1 yarn     yarn        27084 Apr 23 05:17 ./tezlib/lib/jackson-xc-1.9.13.jar\n",
      "655573  132 -r-xr-xr-x   1 yarn     yarn       134133 Apr 23 08:04 ./tezlib/lib/servlet-api-2.5-20081211.jar\n",
      "655492  280 -r-xr-xr-x   1 yarn     yarn       284220 Apr 23 05:14 ./tezlib/lib/commons-lang-2.6.jar\n",
      "655488  480 -r-xr-xr-x   1 yarn     yarn       489884 Apr 23 05:17 ./tezlib/lib/log4j-1.2.17.jar\n",
      "655520   32 -r-xr-xr-x   1 yarn     yarn        29555 Apr 23 05:17 ./tezlib/lib/paranamer-2.3.jar\n",
      "655556 1204 -r-xr-xr-x   1 yarn     yarn      1229125 Apr 23 05:14 ./tezlib/lib/xercesImpl-2.9.1.jar\n",
      "655507   64 -r-xr-xr-x   1 yarn     yarn        62983 Apr 23 05:17 ./tezlib/lib/activation-1.1.jar\n",
      "655532   80 -r-xr-xr-x   1 yarn     yarn        79912 Apr 23 08:05 ./tezlib/lib/api-util-1.0.0-M20.jar\n",
      "655482    8 -r-xr-xr-x   1 yarn     yarn         7784 Apr 25 15:41 ./tezlib/tez-yarn-timeline-history-with-acls-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655471  868 -r-xr-xr-x   1 yarn     yarn       888761 Apr 25 15:38 ./tezlib/tez-api-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655477 1196 -r-xr-xr-x   1 yarn     yarn      1222496 Apr 25 15:39 ./tezlib/tez-dag-XXX.XXX.XXX.2.4.2.0-258.jar\n",
      "655479 1600 -r-xr-xr-x   1 yarn     yarn      1636310 Apr 25 15:40 ./tezlib/tez-ui-XXX.XXX.XXX.2.4.2.0-258.war\n",
      "655680    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor_session.sh.crc\n",
      "655657    4 -r-x------   1 yarn     yarn         1467 Aug 19 10:37 ./job.split\n",
      "655677    8 -rwx------   1 yarn     yarn         4953 Aug 19 10:37 ./launch_container.sh\n",
      "655679    4 -rwx------   1 yarn     yarn          728 Aug 19 10:37 ./default_container_executor_session.sh\n",
      "655663    4 -r-x------   1 yarn     yarn          357 Aug 19 10:37 ./job.splitmetainfo\n",
      "655660   44 -r-x------   1 yarn     yarn        44350 Aug 19 10:37 ./tez-conf.pb\n",
      "655682    4 -rw-r--r--   1 yarn     yarn           16 Aug 19 10:37 ./.default_container_executor.sh.crc\n",
      "655674    4 drwx--x---   2 yarn     yarn         4096 Aug 19 10:37 ./tmp\n",
      "655654  268 -r-x------   1 yarn     yarn       273458 Aug 19 10:37 ./job.jar\n",
      "broken symlinks(find -L . -maxdepth 5 -type l -ls):\n",
      "End of LogType:directory.info\n",
      "\n",
      "LogType:launch_container.sh\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:4953\n",
      "Log Contents:\n",
      "#!/bin/bash\n",
      "\n",
      "export SHELL=\"/bin/bash\"\n",
      "export NM_HTTP_PORT=\"8042\"\n",
      "export LOCAL_DIRS=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002\"\n",
      "export JAVA_HOME=\"/usr/java/jdk1.7.0_75\"\n",
      "export NM_AUX_SERVICE_mapreduce_shuffle=\"AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\n",
      "\"\n",
      "export HADOOP_YARN_HOME=\"/usr/hdp/XXX.XXX.XXX.0-258/hadoop-yarn\"\n",
      "export CLASSPATH=\"$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR:\"\n",
      "export HADOOP_TOKEN_FILE_LOCATION=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/container_tokens\"\n",
      "export NM_HOST=\"testvm007\"\n",
      "export HADOOP_ROOT_LOGGER=\"INFO,CLA\"\n",
      "export JVM_PID=\"$$\"\n",
      "export USER=\"ansible\"\n",
      "export PWD=\"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002\"\n",
      "export CONTAINER_ID=\"container_e01_1471559811619_0002_01_000002\"\n",
      "export HOME=\"/home/\"\n",
      "export NM_PORT=\"45454\"\n",
      "export LOGNAME=\"ansible\"\n",
      "export HADOOP_CONF_DIR=\"/usr/hdp/current/hadoop-yarn-nodemanager/../hadoop/conf\"\n",
      "export MALLOC_ARENA_MAX=\"4\"\n",
      "export LD_LIBRARY_PATH=\"$PWD:$HADOOP_COMMON_HOME/lib/native:$PWD\"\n",
      "export LOG_DIRS=\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002\"\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/filecache/10/tez.tar.gz\" \"tezlib\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/11/job.split\" \"job.split\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/14/job.xml\" \"job.xml\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/12/tez-conf.pb\" \"tez-conf.pb\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/13/job.splitmetainfo\" \"job.splitmetainfo\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "ln -sf \"/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/filecache/10/job.jar\" \"job.jar\"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "# Creating copy of launch script\n",
      "cp \"launch_container.sh\" \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/launch_container.sh\"\n",
      "chmod 640 \"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/launch_container.sh\"\n",
      "# Determining directory contents\n",
      "echo \"ls -l:\" 1>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "ls -l 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "echo \"find -L . -maxdepth 5 -ls:\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "find -L . -maxdepth 5 -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "echo \"broken symlinks(find -L . -maxdepth 5 -type l -ls):\" 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "find -L . -maxdepth 5 -type l -ls 1>>\"/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/directory.info\"\n",
      "exec /bin/bash -c \"$JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3840m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002 -Dtez.root.logger=INFO,CLA  -Djava.io.tmpdir=$PWD/tmp org.apache.tez.runtime.task.TezChild XXX.XXX.XXX.73 36714 container_e01_1471559811619_0002_01_000002 application_1471559811619_0002 1 1>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/stdout 2>/var/log/hadoop-yarn/userlogs/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/stderr \"\n",
      "hadoop_shell_errorcode=$?\n",
      "if [ $hadoop_shell_errorcode -ne 0 ]\n",
      "then\n",
      "  exit $hadoop_shell_errorcode\n",
      "fi\n",
      "End of LogType:launch_container.sh\n",
      "\n",
      "LogType:stderr\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:564\n",
      "Log Contents:\n",
      "2016-08-19 10:37:10 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12 Completed running task attempt: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13 Completed running task attempt: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:22 Completed running task attempt: attempt_1471559811619_0002_1_00_000004_0\n",
      "End of LogType:stderr\n",
      "\n",
      "LogType:stdout\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:564\n",
      "Log Contents:\n",
      "2016-08-19 10:37:10 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12 Completed running task attempt: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13 Completed running task attempt: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13 Starting to run new task attempt: attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:22 Completed running task attempt: attempt_1471559811619_0002_1_00_000004_0\n",
      "End of LogType:stdout\n",
      "\n",
      "LogType:syslog\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:2158\n",
      "Log Contents:\n",
      "2016-08-19 10:37:08,969 [INFO] [main] |task.TezChild|: TezChild starting with PID=18075, containerIdentifier=container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:09,944 [WARN] [main] |util.NativeCodeLoader|: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "2016-08-19 10:37:10,296 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:10,770 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:10,773 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12,628 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:12,647 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000002: 17 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:12,647 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13,848 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000002\n",
      "2016-08-19 10:37:13,852 [INFO] [TezChild] |task.ContainerReporter|: Got TaskUpdate for containerId= container_e01_1471559811619_0002_01_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:13,853 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:23,040 [INFO] [TezChild] |task.ContainerReporter|: Attempting to fetch new task for container container_e01_1471559811619_0002_01_000002\n",
      "End of LogType:syslog\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000000_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:8586\n",
      "Log Contents:\n",
      "2016-08-19 10:37:10,778 [INFO] [main] |task.TezChild|: Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=1\n",
      "2016-08-19 10:37:10,799 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:10,799 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000000_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:10,801 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:10,824 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:10,985 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000000_0\n",
      "2016-08-19 10:37:10,985 [INFO] [I/O Setup 0 Initialize: {MRInput}] |counters.Limits|: Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120\n",
      "2016-08-19 10:37:10,996 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:10,996 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:11,336 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:11,336 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:11,336 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:11,336 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:11,339 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:11,443 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:11,452 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:11,453 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:11,453 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:11,453 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:11,454 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Input: MRInput being auto started by the framework. Subsequent instances will not be auto-started\n",
      "2016-08-19 10:37:11,454 [INFO] [I/O Setup 0 Start: {MRInput}] |runtime.LogicalIOProcessorRuntimeTask|: Started Input with src edge: MRInput\n",
      "2016-08-19 10:37:11,454 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 1\n",
      "2016-08-19 10:37:11,454 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 IOs to start\n",
      "2016-08-19 10:37:11,454 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:11,456 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:11,456 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000000_0_10001\n",
      "2016-08-19 10:37:11,484 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@1311dbb9, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@79705876, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:11,488 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:11,523 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:11,538 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:11,548 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/job.split\n",
      "2016-08-19 10:37:12,499 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:12,501 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part0:0+118\n",
      "2016-08-19 10:37:12,507 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000000_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:12,507 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0\n",
      "2016-08-19 10:37:12,514 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:12,514 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:12,514 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:12,522 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:12,522 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000000_0_10003_0/file.out\n",
      "2016-08-19 10:37:12,555 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:12,565 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000000_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:12,616 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000000_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1849, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=250, CPU_MILLISECONDS=2520, PHYSICAL_MEMORY_BYTES=290979840, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=290979840, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:12,627 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000000_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000003_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:7806\n",
      "Log Contents:\n",
      "2016-08-19 10:37:12,651 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:12,651 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000003_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:12,652 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:12,652 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:12,689 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000003_0\n",
      "2016-08-19 10:37:12,690 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:12,690 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:12,699 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:12,699 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:12,699 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:12,700 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:12,700 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:12,701 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:12,701 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:12,705 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:12,705 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000003_0_10004\n",
      "2016-08-19 10:37:12,710 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@38b9f79a, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@79705876, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:12,711 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:13,691 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:13,693 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:13,698 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/job.split\n",
      "2016-08-19 10:37:13,728 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:13,729 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part3:0+118\n",
      "2016-08-19 10:37:13,738 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000003_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:13,738 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0\n",
      "2016-08-19 10:37:13,738 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:13,739 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:13,739 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:13,740 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:13,740 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000003_0_10006_0/file.out\n",
      "2016-08-19 10:37:13,742 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:13,743 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000003_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:13,747 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000003_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1411, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=850, PHYSICAL_MEMORY_BYTES=396361728, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=396361728, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:13,848 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000003_0\n",
      "\n",
      "LogType:syslog_attempt_1471559811619_0002_1_00_000004_0\n",
      "Log Upload Time:Fri Aug 19 10:37:29 +0900 2016\n",
      "LogLength:7807\n",
      "Log Contents:\n",
      "2016-08-19 10:37:13,855 [INFO] [main] |metrics.TaskCounterUpdater|: Using ResourceCalculatorProcessTree : org.apache.tez.util.TezMxBeanResourceCalculator\n",
      "2016-08-19 10:37:13,855 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|: Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : QuasiMonteCarlo, VertexName: initialmap, VertexParallelism: 10, TaskAttemptID:attempt_1471559811619_0002_1_00_000004_0, processorName=org.apache.tez.mapreduce.processor.map.MapProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=MRInput, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInputLegacy }}, ], outputSpecList=[{{ destinationVertexName=finalreduce, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]\n",
      "2016-08-19 10:37:13,856 [INFO] [main] |resources.MemoryDistributor|: InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=3579314176, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor\n",
      "2016-08-19 10:37:13,856 [INFO] [TezChild] |task.TezTaskRunner|: Initializing task, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:13,873 [INFO] [TezChild] |processor.MRTask|: MRTask.inited: taskAttemptId = attempt_1471559811619_0002_m_000004_0\n",
      "2016-08-19 10:37:13,879 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor\n",
      "2016-08-19 10:37:13,879 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish\n",
      "2016-08-19 10:37:13,886 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInput|: MRInput using newmapreduce API=true, split via event=false, numPhysicalInputs=1\n",
      "2016-08-19 10:37:13,886 [INFO] [I/O Setup 0 Initialize: {MRInput}] |input.MRInputLegacy|: MRInput MRInputLegacy deferring initialization\n",
      "2016-08-19 10:37:13,886 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish\n",
      "2016-08-19 10:37:13,887 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: All initializers finished\n",
      "2016-08-19 10:37:13,887 [INFO] [TezChild] |resources.MemoryDistributor|: InitialRequests=[MRInput:INPUT:0:org.apache.tez.mapreduce.input.MRInputLegacy], [finalreduce:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]\n",
      "2016-08-19 10:37:13,887 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]\n",
      "2016-08-19 10:37:13,887 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996\n",
      "2016-08-19 10:37:13,887 [INFO] [TezChild] |resources.WeightedScalingMemoryDistributor|: Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 3579314176, TotalAvailable: 2398140497, TotalRequested/TotalJVMHeap:0.03\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |resources.MemoryDistributor|: Allocations=[MRInput:org.apache.tez.mapreduce.input.MRInputLegacy:INPUT:0:0], [finalreduce:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Starting Inputs/Outputs\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Num IOs determined for AutoStart: 0\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: AutoStartComplete\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |task.TezTaskRunner|: Running task, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:13,888 [INFO] [TezChild] |map.MapProcessor|: Running map: attempt_1471559811619_0002_1_00_000004_0_10007\n",
      "2016-08-19 10:37:13,891 [INFO] [TezChild] |impl.ExternalSorter|: finalreduce using: memoryMb=100, keySerializerClass=class org.apache.hadoop.io.BooleanWritable, valueSerializerClass=org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer@3b1b6813, comparator=org.apache.hadoop.io.BooleanWritable$Comparator@79705876, partitioner=org.apache.tez.mapreduce.partition.MRPartitioner, serialization=org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization\n",
      "2016-08-19 10:37:13,891 [INFO] [TezChild] |impl.PipelinedSorter|: Setting up PipelinedSorter for finalreduce: #blocks=1, maxMemUsage=104857600, BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100, UsingHashComparator=false\n",
      "2016-08-19 10:37:22,694 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: reserved.remaining()=104857600, reserved.metasize=16777216\n",
      "2016-08-19 10:37:22,696 [INFO] [TezChild] |split.SplitMetaInfoReaderTez|: Attempting to find splits in dir: .\n",
      "2016-08-19 10:37:22,698 [INFO] [TezChild] |lib.MRInputUtils|: Reading input split file from : file:/hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/container_e01_1471559811619_0002_01_000002/job.split\n",
      "2016-08-19 10:37:22,915 [INFO] [TezChild] |input.MRInput|: Initialized MRInput: MRInput\n",
      "2016-08-19 10:37:22,924 [INFO] [TezChild] |map.MapProcessor|: Processing mapreduce split: hdfs://hdfs-cluster/user/ansible/QuasiMonteCarlo_1471570610739_682679022/in/part4:0+118\n",
      "2016-08-19 10:37:22,927 [INFO] [TezChild] |processor.MRTask|: Task:attempt_1471559811619_0002_m_000004_0 is done. And is in the process of committing\n",
      "2016-08-19 10:37:22,928 [INFO] [TezChild] |task.TezTaskRunner|: Closing task, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0\n",
      "2016-08-19 10:37:22,928 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Starting flush of map output\n",
      "2016-08-19 10:37:22,928 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Span0.length = 2, perItem = 9\n",
      "2016-08-19 10:37:22,928 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: done sorting span=0, length=2, time=0\n",
      "2016-08-19 10:37:22,929 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Heap = SpanIterator<0:1> (span=Span[16777216,18]),\n",
      "2016-08-19 10:37:22,929 [INFO] [TezChild] |impl.PipelinedSorter|: finalreduce: Spilling to /hadoop/tmp/hadoop-yarn/nm-local-dir/usercache/ansible/appcache/application_1471559811619_0002/output/attempt_1471559811619_0002_1_00_000004_0_10009_0/file.out\n",
      "2016-08-19 10:37:22,931 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Joining on EventRouter\n",
      "2016-08-19 10:37:22,933 [INFO] [TezChild] |task.TezTaskRunner|: Task completed, taskAttemptId=attempt_1471559811619_0002_1_00_000004_0, fatalErrorOccurred=false\n",
      "2016-08-19 10:37:22,939 [INFO] [TezChild] |runtime.LogicalIOProcessorRuntimeTask|: Final Counters for attempt_1471559811619_0002_1_00_000004_0: Counters: 26 [[File System Counters FILE_BYTES_READ=1265, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, HDFS_BYTES_READ=118, HDFS_BYTES_WRITTEN=0, HDFS_READ_OPS=2, HDFS_LARGE_READ_OPS=0, HDFS_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPLIT_RAW_BYTES=146, SPILLED_RECORDS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=6640, PHYSICAL_MEMORY_BYTES=501743616, VIRTUAL_MEMORY_BYTES=4630061056, COMMITTED_HEAP_BYTES=501743616, INPUT_RECORDS_PROCESSED=1, OUTPUT_RECORDS=2, OUTPUT_BYTES=18, OUTPUT_BYTES_WITH_OVERHEAD=28, OUTPUT_BYTES_PHYSICAL=32, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1]]\n",
      "2016-08-19 10:37:23,040 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: \n",
      "End of LogType:syslog_attempt_1471559811619_0002_1_00_000004_016/08/19 10:41:50 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -a 'yarn logs -applicationId {yarn_application_id}' -l { target_group } hadoop_client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc69188-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc692dc-495a-11e8-a14e-0242ac130002",
     "previous": "9bc6903e-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "## Swimlanesによる可視化\n",
    "\n",
    "SwimlaneインストールNotebookによりTez-toolsをインストールしていれば、Swimlaneにより可視化することもできる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc692dc-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69430-495a-11e8-a14e-0242ac130002",
     "previous": "9bc69188-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/tmpgwiJOL/swimlane.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile {work_dir}/swimlane.yml\n",
    "- hosts: hadoop_swimlane\n",
    "  tasks:\n",
    "    - name: delete_swimlane_old_output\n",
    "      become: yes\n",
    "      file: path={{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/{{ yarn_application_id }}.svg state=absent\n",
    "  \n",
    "    - name: create_swimlane_analyze\n",
    "      shell: bash yarn-swimlanes.sh {{ yarn_application_id }}\n",
    "      args:\n",
    "        chdir: \"{{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/\"\n",
    "  \n",
    "    - name: download_svg\n",
    "      fetch: src={{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/{{ yarn_application_id }}.svg dest={{ local_svg_path }} flat=yes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc69430-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc6957a-495a-11e8-a14e-0242ac130002",
     "previous": "9bc692dc-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "SVGは一時ディレクトリにダウンロードし、Notebookにレンダリングすることにする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9bc6957a-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc696c4-495a-11e8-a14e-0242ac130002",
     "previous": "9bc69430-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "local_svg_path = os.path.join(work_dir, 'result.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc696c4-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc6980e-495a-11e8-a14e-0242ac130002",
     "previous": "9bc6957a-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "swimlaneのスクリプトを実行し、SVGファイルをダウンロードする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc6980e-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69958-495a-11e8-a14e-0242ac130002",
     "previous": "9bc696c4-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_swimlane] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [delete_swimlane_old_output] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [create_swimlane_analyze] *************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [download_svg] ************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m4\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "extra_vars = ['-e yarn_application_id={}'.format(yarn_application_id), '-e local_svg_path={}'.format(local_svg_path)]\n",
    "!ansible-playbook {' '.join(extra_vars)} {work_dir}/swimlane.yml -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc69958-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69ab6-495a-11e8-a14e-0242ac130002",
     "previous": "9bc6980e-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "ダウンロードしたSVGファイルをレンダリングする。\n",
    "\n",
    "(以下にSwimlanesの出力画像が表示されない場合、JupyterのFileメニューからTrust Notebookすることで出力画像が表示されるようになります。)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc69ab6-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69c00-495a-11e8-a14e-0242ac130002",
     "previous": "9bc69958-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"328\" version=\"1.1\" width=\"2524\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "\t\t<script type=\"text/ecmascript\" xlink:href=\"http://code.jquery.com/jquery-2.1.1.min.js\"/>\n",
       "\t\t<text style=\"font-size: 32px; text-anchor: middle\" transform=\"\" x=\"1034\" y=\"32\">appattempt_1471559811619_0002_000001</text><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"84\" y=\"96\">Container ID</text><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"96\" y=\"152\">container_e01_1471559811619_0002_01_000003</text><line style=\"stroke: #ccc\" x1=\"100\" x2=\"2168\" y1=\"152\" y2=\"152\"/><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"96\" y=\"176\">container_e01_1471559811619_0002_01_000002</text><line style=\"stroke: #ccc\" x1=\"100\" x2=\"2168\" y1=\"176\" y2=\"176\"/><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"96\" y=\"200\">container_e01_1471559811619_0002_01_000004</text><line style=\"stroke: #ccc\" x1=\"100\" x2=\"2168\" y1=\"200\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"100\" y=\"116\">0.00 s</text><line style=\"stroke: #ddd\" x1=\"100\" x2=\"100\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"550\" y=\"116\">4.00 s</text><line style=\"stroke: #ddd\" x1=\"550\" x2=\"550\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1000\" y=\"116\">8.00 s</text><line style=\"stroke: #ddd\" x1=\"1000\" x2=\"1000\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1450\" y=\"116\">12.00 s</text><line style=\"stroke: #ddd\" x1=\"1450\" x2=\"1450\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1900\" y=\"116\">16.00 s</text><line style=\"stroke: #ddd\" x1=\"1900\" x2=\"1900\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"370\" y=\"116\">2.00 s</text><line style=\"stroke: #ddd\" x1=\"370\" x2=\"370\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"820\" y=\"116\">6.00 s</text><line style=\"stroke: #ddd\" x1=\"820\" x2=\"820\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1270\" y=\"116\">10.00 s</text><line style=\"stroke: #ddd\" x1=\"1270\" x2=\"1270\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1720\" y=\"116\">14.00 s</text><line style=\"stroke: #ddd\" x1=\"1720\" x2=\"1720\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"190\" y=\"116\">0.00 s</text><line style=\"stroke: #ddd\" x1=\"190\" x2=\"190\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"640\" y=\"116\">4.00 s</text><line style=\"stroke: #ddd\" x1=\"640\" x2=\"640\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1090\" y=\"116\">8.00 s</text><line style=\"stroke: #ddd\" x1=\"1090\" x2=\"1090\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1540\" y=\"116\">12.00 s</text><line style=\"stroke: #ddd\" x1=\"1540\" x2=\"1540\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1990\" y=\"116\">17.00 s</text><line style=\"stroke: #ddd\" x1=\"1990\" x2=\"1990\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"460\" y=\"116\">3.00 s</text><line style=\"stroke: #ddd\" x1=\"460\" x2=\"460\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"910\" y=\"116\">7.00 s</text><line style=\"stroke: #ddd\" x1=\"910\" x2=\"910\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1360\" y=\"116\">11.00 s</text><line style=\"stroke: #ddd\" x1=\"1360\" x2=\"1360\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1810\" y=\"116\">15.00 s</text><line style=\"stroke: #ddd\" x1=\"1810\" x2=\"1810\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"280\" y=\"116\">1.00 s</text><line style=\"stroke: #ddd\" x1=\"280\" x2=\"280\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"730\" y=\"116\">5.00 s</text><line style=\"stroke: #ddd\" x1=\"730\" x2=\"730\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1180\" y=\"116\">9.00 s</text><line style=\"stroke: #ddd\" x1=\"1180\" x2=\"1180\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"2168\" y=\"116\">18.00 s</text><line style=\"stroke: #ddd\" x1=\"2168\" x2=\"2168\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1630\" y=\"116\">13.00 s</text><line style=\"stroke: #ddd\" x1=\"1630\" x2=\"1630\" y1=\"116\" y2=\"200\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"2080\" y=\"116\">17.00 s</text><line style=\"stroke: #ddd\" x1=\"2080\" x2=\"2080\" y1=\"116\" y2=\"200\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"2168\" y1=\"128\" y2=\"128\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"2168\" y1=\"200\" y2=\"200\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"100\" y1=\"128\" y2=\"200\"/><line style=\"stroke: #000\" x1=\"2168\" x2=\"2168\" y1=\"128\" y2=\"200\"/><line style=\"stroke: green\" x1=\"469\" x2=\"469\" y1=\"176\" y2=\"200\"/><line style=\"stroke: green\" x1=\"2284\" x2=\"2284\" y1=\"176\" y2=\"200\"/><rect height=\"24\" style=\"fill: #ccc; opacity: 0.3\" width=\"1815\" x=\"469\" y=\"176\"><title/></rect><line style=\"stroke: green\" x1=\"469\" x2=\"469\" y1=\"128\" y2=\"152\"/><line style=\"stroke: green\" x1=\"2756\" x2=\"2756\" y1=\"128\" y2=\"152\"/><rect height=\"24\" style=\"fill: #ccc; opacity: 0.3\" width=\"2287\" x=\"469\" y=\"128\"><title/></rect><line style=\"stroke: green\" x1=\"469\" x2=\"469\" y1=\"152\" y2=\"176\"/><line style=\"stroke: green\" x1=\"2756\" x2=\"2756\" y1=\"152\" y2=\"176\"/><rect height=\"24\" style=\"fill: #ccc; opacity: 0.3\" width=\"2287\" x=\"469\" y=\"152\"><title/></rect><line stroke-dasharray=\"8,4\" style=\"stroke: black;\" x1=\"198\" x2=\"198\" y1=\"104\" y2=\"200\"/><line stroke-dasharray=\"8,4\" style=\"stroke: black;\" x1=\"2191\" x2=\"2191\" y1=\"104\" y2=\"200\"/><line style=\"stroke: black\" x1=\"198\" x2=\"2191\" y1=\"104\" y2=\"104\"/><text style=\"text-anchor: middle; font-size: 12px;\" transform=\"\" x=\"1194\" y=\"96\">dag_1471559811619_0002_1 (17.9 s)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"225\" x=\"737\" y=\"153\"><title>attempt_1471559811619_0002_1_00_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"849\" y=\"163\">initialmap (00000_0)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"123\" x=\"965\" y=\"153\"><title>attempt_1471559811619_0002_1_00_000003_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1026\" y=\"163\">initialmap (00003_0)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"311\" x=\"793\" y=\"129\"><title>attempt_1471559811619_0002_1_00_000001_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"948\" y=\"139\">initialmap (00001_0)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"16\" x=\"1106\" y=\"129\"><title>attempt_1471559811619_0002_1_00_000005_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1114\" y=\"139\">initialmap</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"287\" x=\"839\" y=\"177\"><title>attempt_1471559811619_0002_1_00_000002_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"982\" y=\"187\">initialmap (00002_0)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"18\" x=\"1127\" y=\"177\"><title>attempt_1471559811619_0002_1_00_000008_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1136\" y=\"187\">initialmap</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"17\" x=\"1134\" y=\"129\"><title>attempt_1471559811619_0002_1_00_000006_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1142\" y=\"139\">initialmap</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"19\" x=\"1163\" y=\"129\"><title>attempt_1471559811619_0002_1_00_000007_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1172\" y=\"139\">initialmap</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"404\" x=\"1156\" y=\"177\"><title>attempt_1471559811619_0002_1_00_000009_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1358\" y=\"187\">initialmap (00009_0)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"1010\" x=\"1099\" y=\"153\"><title>attempt_1471559811619_0002_1_00_000004_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1604\" y=\"163\">initialmap (00004_0)</text><rect height=\"22\" style=\"fill: #62C2A2; stroke: #ccc;\" width=\"974\" x=\"1194\" y=\"129\"><title>attempt_1471559811619_0002_1_01_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1681\" y=\"139\">finalreduce (00000_0)</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename=local_svg_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9bc69c00-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69d4a-495a-11e8-a14e-0242ac130002",
     "previous": "9bc69ab6-495a-11e8-a14e-0242ac130002"
    }
   },
   "source": [
    "# 後始末\n",
    "\n",
    "一時ディレクトリを削除する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9bc69d4a-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": "9bc69e94-495a-11e8-a14e-0242ac130002",
     "previous": "9bc69c00-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "!rm -fr {work_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9bc69e94-495a-11e8-a14e-0242ac130002",
     "history": [],
     "next": null,
     "previous": "9bc69d4a-495a-11e8-a14e-0242ac130002"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "lc_notebook_meme": {
   "current": "9bc67d2e-495a-11e8-a14e-0242ac130002",
   "history": [],
   "root_cells": [
    "9bc67fe0-495a-11e8-a14e-0242ac130002",
    "9bc68184-495a-11e8-a14e-0242ac130002",
    "9bc682ec-495a-11e8-a14e-0242ac130002",
    "9bc6844a-495a-11e8-a14e-0242ac130002",
    "9bc685a8-495a-11e8-a14e-0242ac130002",
    "9bc686f2-495a-11e8-a14e-0242ac130002",
    "9bc68850-495a-11e8-a14e-0242ac130002",
    "9bc689a4-495a-11e8-a14e-0242ac130002",
    "9bc68aee-495a-11e8-a14e-0242ac130002",
    "9bc68c42-495a-11e8-a14e-0242ac130002",
    "9bc68d96-495a-11e8-a14e-0242ac130002",
    "9bc68eea-495a-11e8-a14e-0242ac130002",
    "9bc6903e-495a-11e8-a14e-0242ac130002",
    "9bc69188-495a-11e8-a14e-0242ac130002",
    "9bc692dc-495a-11e8-a14e-0242ac130002",
    "9bc69430-495a-11e8-a14e-0242ac130002",
    "9bc6957a-495a-11e8-a14e-0242ac130002",
    "9bc696c4-495a-11e8-a14e-0242ac130002",
    "9bc6980e-495a-11e8-a14e-0242ac130002",
    "9bc69958-495a-11e8-a14e-0242ac130002",
    "9bc69ab6-495a-11e8-a14e-0242ac130002",
    "9bc69c00-495a-11e8-a14e-0242ac130002",
    "9bc69d4a-495a-11e8-a14e-0242ac130002",
    "9bc69e94-495a-11e8-a14e-0242ac130002"
   ]
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}